Sep 16 13:41:03.447655 ip-10-0-139-34 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 13:41:03.447667 ip-10-0-139-34 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 13:41:03.447674 ip-10-0-139-34 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 13:41:03.447870 ip-10-0-139-34 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 13:41:13.696897 ip-10-0-139-34 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 13:41:13.696918 ip-10-0-139-34 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot eb9f8bf604a74535a850046123ae9a5f -- Sep 16 13:43:00.840237 ip-10-0-139-34 systemd[1]: Starting Kubernetes Kubelet... Sep 16 13:43:01.275666 ip-10-0-139-34 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:43:01.275666 ip-10-0-139-34 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 13:43:01.275666 ip-10-0-139-34 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:43:01.275666 ip-10-0-139-34 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 13:43:01.275666 ip-10-0-139-34 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:43:01.276770 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.276687 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 13:43:01.280062 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280045 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.280062 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280061 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280065 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280069 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280072 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280076 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280079 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280087 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.280164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280090 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.280819 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280810 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.280819 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280818 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280822 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280825 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280829 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280832 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280835 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280839 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280842 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280844 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280847 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280850 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280852 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280854 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280857 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280860 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280862 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280876 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280879 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280882 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.280885 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280885 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280888 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280891 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280894 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280896 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280900 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280902 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280905 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280907 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280910 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280913 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280915 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280918 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280920 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280923 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280925 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280930 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280935 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280938 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.281352 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280941 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280944 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280947 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280950 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280954 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280957 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280959 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280962 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280965 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280967 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280970 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280972 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280979 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280981 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280984 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280987 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280990 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280992 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280995 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.280997 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.281818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281000 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281003 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281005 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281008 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281011 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281014 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281016 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281019 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281021 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281023 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281026 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281028 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281031 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281033 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281036 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281038 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281041 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281043 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281046 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281049 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.282312 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281052 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281055 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281057 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281060 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281062 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281065 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281068 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281073 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281076 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281078 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281081 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281800 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281806 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281809 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281813 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281816 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281820 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281824 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281827 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.282792 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281830 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281833 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281835 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281838 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281840 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281844 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281846 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281849 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281851 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281854 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281857 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281860 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281862 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281879 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281882 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281884 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281887 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281889 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281892 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281895 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.283243 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281898 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281900 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281903 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281905 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281908 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281910 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281913 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281916 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281919 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281922 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281927 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281929 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281932 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281934 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281937 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281939 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281942 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281944 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281947 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281950 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.283756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281953 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281957 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281961 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281964 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281967 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281971 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281974 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281977 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281980 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281983 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281986 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281989 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281992 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281995 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.281998 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282000 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282003 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282006 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282008 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.284237 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282011 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282014 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282016 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282018 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282022 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282025 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282027 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282030 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282032 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282034 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282037 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282039 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282044 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282048 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282051 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282054 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282057 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282060 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282063 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.284696 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282066 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282069 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282072 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282074 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282077 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282079 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282082 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282084 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282087 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282089 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282092 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282094 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282097 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282481 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282486 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282489 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282492 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282495 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282498 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282502 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.285161 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282504 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282507 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282509 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282512 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282515 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282518 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282520 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282525 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282528 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282547 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282551 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282555 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282558 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282561 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282564 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282566 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282569 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282572 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282575 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282578 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.285677 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282580 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282583 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282586 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282588 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282591 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282593 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282596 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282598 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282601 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282603 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282606 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282610 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282613 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282616 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282619 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282621 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282623 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282626 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282628 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.286182 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282631 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282633 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282637 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282640 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282644 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282646 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282649 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282652 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282654 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282657 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282659 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282662 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282665 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282667 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282671 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282673 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282676 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282678 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282681 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.286678 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282683 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282686 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282688 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282691 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282693 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282696 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282699 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282701 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282704 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282706 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282708 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282711 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282713 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282716 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282718 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282722 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282724 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282728 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282731 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282735 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.287149 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282738 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282741 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282744 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282747 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282749 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282752 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282755 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282757 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282760 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282762 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282765 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282767 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282770 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.282772 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282841 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282893 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282900 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282905 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282910 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282914 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282918 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 13:43:01.287701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282922 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282925 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282927 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282931 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282934 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282937 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282942 2559 flags.go:64] FLAG: --cgroup-root="" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282944 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282949 2559 flags.go:64] FLAG: --client-ca-file="" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282952 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282955 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282960 2559 flags.go:64] FLAG: --cluster-domain="" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282963 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282966 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282969 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282972 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282976 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282979 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282982 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282985 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282988 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282991 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282994 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.282997 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 13:43:01.288204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283000 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283004 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283007 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283010 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283013 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283015 2559 flags.go:64] FLAG: --enable-server="true" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283019 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283023 2559 flags.go:64] FLAG: --event-burst="100" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283026 2559 flags.go:64] FLAG: --event-qps="50" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283029 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283032 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283034 2559 flags.go:64] FLAG: --eviction-hard="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283043 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283045 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283050 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283053 2559 flags.go:64] FLAG: --eviction-soft="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283057 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283060 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283063 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283065 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283068 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283071 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283074 2559 flags.go:64] FLAG: --feature-gates="" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283078 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283081 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 13:43:01.288798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283084 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283087 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283090 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283093 2559 flags.go:64] FLAG: --help="false" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283095 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283099 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283101 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283104 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283108 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283111 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283114 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283116 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283119 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283122 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283125 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283128 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283130 2559 flags.go:64] FLAG: --kube-reserved="" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283133 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283136 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283139 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283142 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283145 2559 flags.go:64] FLAG: --lock-file="" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283148 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283152 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 13:43:01.289440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283155 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283160 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283163 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283165 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283168 2559 flags.go:64] FLAG: --logging-format="text" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283171 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283174 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283177 2559 flags.go:64] FLAG: --manifest-url="" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283180 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283184 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283186 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283190 2559 flags.go:64] FLAG: --max-pods="110" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283193 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283196 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283199 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283201 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283204 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283207 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283210 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283217 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283220 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283223 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283226 2559 flags.go:64] FLAG: --pod-cidr="" Sep 16 13:43:01.290058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283229 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283234 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283237 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283240 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283243 2559 flags.go:64] FLAG: --port="10250" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283246 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283250 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a3eb3885af968476" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283254 2559 flags.go:64] FLAG: --qos-reserved="" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283257 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283261 2559 flags.go:64] FLAG: --register-node="true" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283263 2559 flags.go:64] FLAG: --register-with-taints="" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283268 2559 flags.go:64] FLAG: --registry-burst="10" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283271 2559 flags.go:64] FLAG: --registry-qps="5" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283273 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283276 2559 flags.go:64] FLAG: --reserved-memory="" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283279 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283282 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283285 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283288 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283291 2559 flags.go:64] FLAG: --runonce="false" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283294 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283297 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283300 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283302 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283305 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283308 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 13:43:01.290676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283311 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283314 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283317 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283320 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283323 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283326 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283329 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283332 2559 flags.go:64] FLAG: --system-cgroups="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283335 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283340 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283343 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283345 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283351 2559 flags.go:64] FLAG: --tls-min-version="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283353 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283356 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283359 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283363 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283366 2559 flags.go:64] FLAG: --v="2" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283370 2559 flags.go:64] FLAG: --version="false" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283374 2559 flags.go:64] FLAG: --vmodule="" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283377 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.283380 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283493 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283497 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283500 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.291305 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283503 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283506 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283509 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283512 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283514 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283517 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283520 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283523 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283526 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283528 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283531 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283533 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283536 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283538 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283541 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283544 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283547 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283549 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283552 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283568 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.291938 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283572 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283575 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283578 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283581 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283584 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283587 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283589 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283592 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283595 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283597 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283600 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283602 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283605 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283607 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283610 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283612 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283614 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283617 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283619 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.292437 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283622 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283624 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283627 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283630 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283634 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283637 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283640 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283642 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283645 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283648 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283650 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283653 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283656 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283659 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283661 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283664 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283667 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283669 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283672 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.292910 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283674 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283677 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283681 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283684 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283687 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283690 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283693 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283695 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283698 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283700 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283703 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283706 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283708 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283710 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283713 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283715 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283718 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283720 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283723 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.293382 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283726 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283729 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283732 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283734 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283737 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283740 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283742 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283745 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283747 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283752 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283754 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283757 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283759 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283771 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283773 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283776 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283778 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283781 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.293848 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.283784 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.284409 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.292494 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.292508 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.293404 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.293416 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293484 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293489 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293492 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293495 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293498 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293502 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293505 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293508 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293511 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.294294 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293513 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293516 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293519 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293531 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293534 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293537 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293540 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293542 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293545 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293548 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293551 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293554 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293556 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293559 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293562 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293564 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293567 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293569 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293572 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293574 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.294676 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293577 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293580 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293582 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293585 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293587 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293590 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293592 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293595 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293597 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293600 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293602 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293605 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293608 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293610 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293613 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293615 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293624 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293626 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293629 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293631 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.295154 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293634 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293637 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293639 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293642 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293644 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293647 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293650 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293652 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293655 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293657 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293660 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293662 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293665 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293667 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293670 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293672 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293675 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293677 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293680 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293682 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.295659 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293685 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293688 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293690 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293693 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293697 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293701 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293705 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293708 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293711 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293720 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293723 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293726 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293729 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293731 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293734 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293737 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293739 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293741 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293744 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.296355 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293746 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293749 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293753 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293756 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293759 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293762 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293764 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293767 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293769 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293772 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293774 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.293779 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293939 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293945 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:43:01.297111 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293948 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293951 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293954 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293957 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293960 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293962 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293966 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293968 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293971 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293980 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293983 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293986 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293988 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293991 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293994 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293996 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.293999 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294002 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294004 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294006 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:43:01.297483 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294009 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294012 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294014 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294017 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294020 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294023 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294025 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294028 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294030 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294034 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294038 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294041 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294044 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294047 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294050 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294053 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294055 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294058 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294061 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:43:01.298011 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294063 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294067 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294070 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294078 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294081 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294084 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294086 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294088 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294091 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294095 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294098 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294101 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294104 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294106 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294109 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294112 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294121 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294123 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294126 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:43:01.298637 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294129 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294131 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294133 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294136 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294139 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294141 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294144 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294147 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294149 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294151 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294154 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294156 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294161 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294163 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294166 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294169 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294171 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294186 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294188 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294191 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:43:01.299105 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294194 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294196 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294199 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294201 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294204 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294207 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294209 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294212 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294214 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294217 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294220 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294222 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294225 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294227 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294230 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294234 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294236 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294239 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:43:01.299617 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:01.294241 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:43:01.300163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.294246 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:43:01.300163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.294351 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 13:43:01.300163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.299133 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 13:43:01.300288 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.300275 2559 server.go:1019] "Starting client certificate rotation" Sep 16 13:43:01.300405 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.300386 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:43:01.301287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.301275 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:43:01.303276 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.303260 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:43:01.304426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.304407 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:43:01.316155 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.316138 2559 log.go:25] "Validated CRI v1 runtime API" Sep 16 13:43:01.319396 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.319380 2559 log.go:25] "Validated CRI v1 image API" Sep 16 13:43:01.320952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.320936 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 13:43:01.324387 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.324370 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cafd0979-a9de-4602-99ce-9d0aca148185:/dev/nvme0n1p4 ccd9dacf-091c-41a8-b605-ced16744dfc3:/dev/nvme0n1p3] Sep 16 13:43:01.324430 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.324387 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 13:43:01.331631 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.331525 2559 manager.go:216] Machine: {Timestamp:2026-09-16 13:43:01.330048607 +0000 UTC m=+0.370642555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3181057 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe136648d521569af40fc796f4e67 SystemUUID:ec2fe136-648d-5215-69af-40fc796f4e67 BootID:eb9f8bf6-04a7-4535-a850-046123ae9a5f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:06:a8:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:06:a8:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:2e:a6:ca:23:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 13:43:01.331631 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.331628 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 13:43:01.331742 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.331704 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 13:43:01.333538 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.333521 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:43:01.335248 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.335222 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 13:43:01.335374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.335250 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 13:43:01.336084 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.336073 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 13:43:01.336121 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.336086 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 13:43:01.336121 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.336095 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:43:01.336121 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.336109 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:43:01.337457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.337447 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 13:43:01.341149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341138 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:43:01.341267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341259 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 13:43:01.341321 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341312 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 16 13:43:01.341376 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341349 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 13:43:01.341376 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341364 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 13:43:01.341433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341378 2559 kubelet.go:398] "Adding apiserver pod source" Sep 16 13:43:01.341433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341390 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 13:43:01.341808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341795 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:43:01.341858 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.341823 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:43:01.344880 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.344853 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 13:43:01.347257 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.347243 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 13:43:01.347327 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.347272 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 13:43:01.349318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349305 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349320 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349345 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349350 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349361 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349368 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349381 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 13:43:01.349386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349386 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 13:43:01.349575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349393 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 13:43:01.349575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349398 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 13:43:01.349575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349406 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 13:43:01.349575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.349412 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 13:43:01.350284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.350274 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 13:43:01.350317 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.350285 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 13:43:01.353460 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.353441 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 13:43:01.354134 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.354123 2559 server.go:1268] "Started kubelet" Sep 16 13:43:01.354273 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.354241 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 13:43:01.354273 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.354246 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 13:43:01.354413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.354277 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 13:43:01.354413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.354206 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 13:43:01.354413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.354339 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 13:43:01.354939 ip-10-0-139-34 systemd[1]: Started Kubernetes Kubelet. Sep 16 13:43:01.356536 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.356509 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 13:43:01.357079 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.357067 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 16 13:43:01.361197 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.361177 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 13:43:01.363014 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.362994 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-664tr" Sep 16 13:43:01.363182 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.362080 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-34.ec2.internal.18d5d0f5dda9c870 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-34.ec2.internal,UID:ip-10-0-139-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-34.ec2.internal,},FirstTimestamp:2026-09-16 13:43:01.354096752 +0000 UTC m=+0.394690698,LastTimestamp:2026-09-16 13:43:01.354096752 +0000 UTC m=+0.394690698,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-34.ec2.internal,}" Sep 16 13:43:01.364270 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.364251 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 13:43:01.365019 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.364998 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 13:43:01.365133 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.365106 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.365777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365761 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 13:43:01.365777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365769 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 13:43:01.365777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365762 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 13:43:01.365777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365780 2559 factory.go:55] Registering systemd factory Sep 16 13:43:01.365987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365787 2559 factory.go:223] Registration of the systemd container factory successfully Sep 16 13:43:01.365987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365797 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 13:43:01.365987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365878 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 16 13:43:01.365987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365886 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 16 13:43:01.366095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.365990 2559 factory.go:153] Registering CRI-O factory Sep 16 13:43:01.366095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.366006 2559 factory.go:223] Registration of the crio container factory successfully Sep 16 13:43:01.366095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.366029 2559 factory.go:103] Registering Raw factory Sep 16 13:43:01.366095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.366043 2559 manager.go:1196] Started watching for new ooms in manager Sep 16 13:43:01.367307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.367219 2559 manager.go:349] Starting recovery of all containers Sep 16 13:43:01.367547 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.367474 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 13:43:01.368350 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.368329 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 13:43:01.372412 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.372393 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-664tr" Sep 16 13:43:01.378480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.378448 2559 manager.go:354] Recovery completed Sep 16 13:43:01.384110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.384096 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.385733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.385719 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.385803 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.385745 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.385803 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.385755 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.386327 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.386310 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 13:43:01.386327 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.386324 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.386423 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.386326 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 13:43:01.386423 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.386364 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:43:01.386992 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.386974 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.387075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.387007 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.387157 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.387023 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.387663 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.387646 2559 policy_none.go:49] "None policy: Start" Sep 16 13:43:01.387747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.387667 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 13:43:01.387747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.387680 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 13:43:01.388838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.388824 2559 policy_none.go:47] "Start" Sep 16 13:43:01.415632 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.415617 2559 manager.go:337] "Starting Device Plugin manager" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.415649 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.415669 2559 server.go:87] "Starting device plugin registration server" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.416002 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.416012 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.416106 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.416174 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.416180 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.416691 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 13:43:01.438932 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.416737 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.463568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.463530 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 13:43:01.464732 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.464709 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 13:43:01.464831 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.464761 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 13:43:01.464831 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.464792 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 13:43:01.464952 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.464903 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 13:43:01.467376 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.467358 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:43:01.516534 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.516509 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.517825 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.517803 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.517932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.517834 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.517932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.517844 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.517932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.517882 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.524558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.524542 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.524628 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.524564 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-34.ec2.internal\": node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.554371 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.554301 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.565399 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.565379 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal"] Sep 16 13:43:01.565456 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.565446 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.566163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.566144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.566285 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.566166 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.566368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.566350 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.566413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.566384 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.566413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.566403 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.568666 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.568654 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.568790 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.568777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.568826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.568804 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.569426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569413 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.569426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569421 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.569524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569438 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.569524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569441 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.569524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569448 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.569524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.569454 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.571663 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.571650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.571720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.571678 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:43:01.572412 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.572399 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:43:01.572480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.572420 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:43:01.572480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.572431 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:43:01.594426 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.594402 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-34.ec2.internal\" not found" node="ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.598736 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.598720 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-34.ec2.internal\" not found" node="ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.655141 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.655119 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.667169 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.667151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.667225 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.667175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.667225 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.667194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90413d152f265712b7c74d8e704e0cb6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-34.ec2.internal\" (UID: \"90413d152f265712b7c74d8e704e0cb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.667290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.667232 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.667290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.667238 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e63221b9a751cb21bdf822925050ce0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal\" (UID: \"6e63221b9a751cb21bdf822925050ce0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.755491 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.755458 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.767821 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.767795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90413d152f265712b7c74d8e704e0cb6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-34.ec2.internal\" (UID: \"90413d152f265712b7c74d8e704e0cb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.767961 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.767832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90413d152f265712b7c74d8e704e0cb6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-34.ec2.internal\" (UID: \"90413d152f265712b7c74d8e704e0cb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.856244 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.856174 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:01.896724 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.896683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.901488 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:01.901412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:01.956832 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:01.956797 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.062582 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.062531 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.170212 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.170126 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.272162 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.272131 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.301248 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.301214 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 13:43:02.301719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.301389 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:43:02.357171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.357149 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:43:02.372537 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.372504 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.379839 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.379807 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:38:01 +0000 UTC" deadline="2028-03-16 04:53:20.660584719 +0000 UTC" Sep 16 13:43:02.379839 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.379837 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13119h10m18.28075106s" Sep 16 13:43:02.473276 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.473192 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.574127 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:02.574094 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-34.ec2.internal\" not found" Sep 16 13:43:02.645494 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.645465 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:43:02.666266 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.666235 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" Sep 16 13:43:02.680194 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.680173 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:43:02.682255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.682239 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" Sep 16 13:43:02.696167 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.696150 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:43:02.753095 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:02.753060 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90413d152f265712b7c74d8e704e0cb6.slice/crio-6358881e8a19a1711543549d297630feca9a6acc126716cec2f599c39f81d63e WatchSource:0}: Error finding container 6358881e8a19a1711543549d297630feca9a6acc126716cec2f599c39f81d63e: Status 404 returned error can't find the container with id 6358881e8a19a1711543549d297630feca9a6acc126716cec2f599c39f81d63e Sep 16 13:43:02.753444 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:02.753423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e63221b9a751cb21bdf822925050ce0.slice/crio-bcaac8c12786ee00e169270881e65316799f6cafbcfb41245f3a7a19c798e401 WatchSource:0}: Error finding container bcaac8c12786ee00e169270881e65316799f6cafbcfb41245f3a7a19c798e401: Status 404 returned error can't find the container with id bcaac8c12786ee00e169270881e65316799f6cafbcfb41245f3a7a19c798e401 Sep 16 13:43:02.760812 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.760785 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:43:02.760812 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:02.760801 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:43:03.344099 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.344068 2559 apiserver.go:52] "Watching apiserver" Sep 16 13:43:03.354528 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.354502 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 13:43:03.354953 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.354929 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-d4v2t","openshift-image-registry/node-ca-bhsxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal","openshift-multus/multus-additional-cni-plugins-5w2tn","openshift-multus/multus-rr9k5","kube-system/global-pull-secret-syncer-8stjr","openshift-dns/node-resolver-zrbdq","openshift-multus/network-metrics-daemon-cwjfb","openshift-network-diagnostics/network-check-target-6d6sc","openshift-network-operator/iptables-alerter-vz94h","openshift-ovn-kubernetes/ovnkube-node-frj4k","kube-system/konnectivity-agent-l67cb","kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w"] Sep 16 13:43:03.357544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.357523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.360274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.359873 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.360274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.360036 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.360274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.360068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mrs6j\"" Sep 16 13:43:03.360274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.360072 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.362508 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.362486 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 13:43:03.362732 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.362714 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.362810 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.362760 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.363374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.363177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.363374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.363193 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 13:43:03.363374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.363229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvl5l\"" Sep 16 13:43:03.363374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.363179 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 13:43:03.363617 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.363485 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 13:43:03.364785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.364763 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 13:43:03.364882 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.364783 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 13:43:03.365045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.365017 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.365138 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.365100 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nslqn\"" Sep 16 13:43:03.365505 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.365478 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.365586 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.365575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.368213 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ddzvl\"" Sep 16 13:43:03.368213 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368210 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 13:43:03.368398 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368253 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 13:43:03.368528 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368511 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 13:43:03.368599 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368525 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.368599 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.368747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.368708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.371106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.371089 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.371203 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.371177 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:03.371403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.371384 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7nvjh\"" Sep 16 13:43:03.371678 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.371660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 13:43:03.373522 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.373504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:03.373610 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.373560 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:03.376430 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.376413 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.376527 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.376491 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:03.377646 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377622 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-socket-dir-parent\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.377754 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-systemd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.377754 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-etc-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.377947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-script-lib\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.377947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377813 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-cnibin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.377947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377848 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-bin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.377947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-daemon-config\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.377947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-multus-certs\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.377999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78jm\" (UniqueName: \"kubernetes.io/projected/272e92b2-262a-4d7d-b2af-38c8f9c2419d-kube-api-access-n78jm\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4hf\" (UniqueName: \"kubernetes.io/projected/6adf8cef-fca8-4a55-be3f-5439845b971a-kube-api-access-sj4hf\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-k8s-cni-cncf-io\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d54b1b-882a-487d-b624-2840f40fb916-tmp-dir\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-node-log\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-netns\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-netd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-netns\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378322 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-kubelet\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378375 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-systemd-units\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378443 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-device-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-hostroot\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kjv\" (UniqueName: \"kubernetes.io/projected/0d63c179-5550-4ee7-b927-bd95005a4d99-kube-api-access-c9kjv\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-kubelet\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-var-lib-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.378621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-env-overrides\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-slash\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378756 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-bin\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-cni-binary-copy\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-multus\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-conf-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tr7g\" (UniqueName: \"kubernetes.io/projected/96d54b1b-882a-487d-b624-2840f40fb916-kube-api-access-9tr7g\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-ovn\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.378993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsb7\" (UniqueName: \"kubernetes.io/projected/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kube-api-access-6dsb7\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-cnibin\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-system-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.379343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-etc-kubernetes\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d54b1b-882a-487d-b624-2840f40fb916-hosts-file\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-log-socket\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-config\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovn-node-metrics-cert\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.380149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.379240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.381628 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.381607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.382403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.382134 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 13:43:03.382403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.382161 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:43:03.382403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.382227 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.382403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.382388 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.382771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.382741 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kj99r\"" Sep 16 13:43:03.384429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.384318 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.387542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.386615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 13:43:03.387542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.387156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xdhdr\"" Sep 16 13:43:03.387542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.387509 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.387684 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.387598 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.389725 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.389682 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.389829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.389751 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:03.389829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.389757 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:03.390053 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.390033 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2b57\"" Sep 16 13:43:03.392407 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.392387 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 13:43:03.392507 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.392389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-958k5\"" Sep 16 13:43:03.392572 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.392552 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 13:43:03.417941 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.417916 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-twdnr" Sep 16 13:43:03.429557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.429536 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-twdnr" Sep 16 13:43:03.467912 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.467886 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 13:43:03.471849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.471801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" event={"ID":"90413d152f265712b7c74d8e704e0cb6","Type":"ContainerStarted","Data":"6358881e8a19a1711543549d297630feca9a6acc126716cec2f599c39f81d63e"} Sep 16 13:43:03.473936 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.473908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" event={"ID":"6e63221b9a751cb21bdf822925050ce0","Type":"ContainerStarted","Data":"bcaac8c12786ee00e169270881e65316799f6cafbcfb41245f3a7a19c798e401"} Sep 16 13:43:03.480489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-konnectivity-ca\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.480489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-systemd-units\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.480489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzjf\" (UniqueName: \"kubernetes.io/projected/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-kube-api-access-6xzjf\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.480489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-systemd-units\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480596 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-conf\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-bin\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-systemd\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-lib-modules\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.480749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480741 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-bin\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsb7\" (UniqueName: \"kubernetes.io/projected/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kube-api-access-6dsb7\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-cnibin\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.480954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-cnibin\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-system-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-etc-kubernetes\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-system-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-host\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-log-socket\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-socket-dir-parent\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-cni-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481251 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfvg\" (UniqueName: \"kubernetes.io/projected/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-kube-api-access-rkfvg\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.481284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-log-socket\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-bin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-socket-dir-parent\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-daemon-config\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-etc-kubernetes\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-bin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d54b1b-882a-487d-b624-2840f40fb916-tmp-dir\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-modprobe-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481638 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78jm\" (UniqueName: \"kubernetes.io/projected/272e92b2-262a-4d7d-b2af-38c8f9c2419d-kube-api-access-n78jm\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsx8\" (UniqueName: \"kubernetes.io/projected/c262271c-d309-4bed-9c9d-093d5335e24e-kube-api-access-2qsx8\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kjv\" (UniqueName: \"kubernetes.io/projected/0d63c179-5550-4ee7-b927-bd95005a4d99-kube-api-access-c9kjv\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-netns\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-netd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.482072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d54b1b-882a-487d-b624-2840f40fb916-tmp-dir\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-netns\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-kubelet-config\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-daemon-config\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.481979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-device-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-hostroot\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-tmp\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-kubelet\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-cni-netd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-var-lib-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-netns\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-kubelet\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.482937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-hostroot\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482296 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482325 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-netns\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-env-overrides\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482380 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-var-lib-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-device-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-run-ovn-kubernetes\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-conf-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tr7g\" (UniqueName: \"kubernetes.io/projected/96d54b1b-882a-487d-b624-2840f40fb916-kube-api-access-9tr7g\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-slash\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-multus-conf-dir\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.483829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-host-slash\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-cni-binary-copy\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-multus\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6adf8cef-fca8-4a55-be3f-5439845b971a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-var-lib-kubelet\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482713 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-cni-multus\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-host-slash\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-ovn\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d54b1b-882a-487d-b624-2840f40fb916-hosts-file\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-ovn\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-iptables-alerter-script\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-env-overrides\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96d54b1b-882a-487d-b624-2840f40fb916-hosts-file\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.482978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-config\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovn-node-metrics-cert\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysconfig\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.484731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-agent-certs\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-systemd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-etc-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-script-lib\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d63c179-5550-4ee7-b927-bd95005a4d99-cni-binary-copy\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-cnibin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-multus-certs\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-serviceca\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-run-systemd\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4hf\" (UniqueName: \"kubernetes.io/projected/6adf8cef-fca8-4a55-be3f-5439845b971a-kube-api-access-sj4hf\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-cnibin\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-k8s-cni-cncf-io\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-sys\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483381 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-etc-openvswitch\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-etc-tuned\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-dbus\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.485636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gsd\" (UniqueName: \"kubernetes.io/projected/dffacf5a-422a-42f5-a9c4-65f769ed48ab-kube-api-access-f6gsd\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-config\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-node-log\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483503 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483488 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-k8s-cni-cncf-io\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-kubelet\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-kubernetes\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-var-lib-kubelet\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-run\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-host\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6adf8cef-fca8-4a55-be3f-5439845b971a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d63c179-5550-4ee7-b927-bd95005a4d99-host-run-multus-certs\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/272e92b2-262a-4d7d-b2af-38c8f9c2419d-node-log\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.486550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.483855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovnkube-script-lib\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.488527 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.488481 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/272e92b2-262a-4d7d-b2af-38c8f9c2419d-ovn-node-metrics-cert\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.499467 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.499436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4hf\" (UniqueName: \"kubernetes.io/projected/6adf8cef-fca8-4a55-be3f-5439845b971a-kube-api-access-sj4hf\") pod \"multus-additional-cni-plugins-5w2tn\" (UID: \"6adf8cef-fca8-4a55-be3f-5439845b971a\") " pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.499467 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.499454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsb7\" (UniqueName: \"kubernetes.io/projected/612e8a01-9d6f-4bf6-b499-b60d90f9bad4-kube-api-access-6dsb7\") pod \"aws-ebs-csi-driver-node-bsw8w\" (UID: \"612e8a01-9d6f-4bf6-b499-b60d90f9bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.500469 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.500445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tr7g\" (UniqueName: \"kubernetes.io/projected/96d54b1b-882a-487d-b624-2840f40fb916-kube-api-access-9tr7g\") pod \"node-resolver-zrbdq\" (UID: \"96d54b1b-882a-487d-b624-2840f40fb916\") " pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.504507 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.504484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kjv\" (UniqueName: \"kubernetes.io/projected/0d63c179-5550-4ee7-b927-bd95005a4d99-kube-api-access-c9kjv\") pod \"multus-rr9k5\" (UID: \"0d63c179-5550-4ee7-b927-bd95005a4d99\") " pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.509848 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.509822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78jm\" (UniqueName: \"kubernetes.io/projected/272e92b2-262a-4d7d-b2af-38c8f9c2419d-kube-api-access-n78jm\") pod \"ovnkube-node-frj4k\" (UID: \"272e92b2-262a-4d7d-b2af-38c8f9c2419d\") " pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.584158 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-host\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584171 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfvg\" (UniqueName: \"kubernetes.io/projected/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-kube-api-access-rkfvg\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-modprobe-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsx8\" (UniqueName: \"kubernetes.io/projected/c262271c-d309-4bed-9c9d-093d5335e24e-kube-api-access-2qsx8\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-kubelet-config\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.584341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-tmp\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-var-lib-kubelet\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-host-slash\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-iptables-alerter-script\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-host\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysconfig\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-agent-certs\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysconfig\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-serviceca\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-kubelet-config\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-sys\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-etc-tuned\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-dbus\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584691 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gsd\" (UniqueName: \"kubernetes.io/projected/dffacf5a-422a-42f5-a9c4-65f769ed48ab-kube-api-access-f6gsd\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.584693 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.584698 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-kubernetes\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-run\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.584771 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.084746729 +0000 UTC m=+3.125340684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584789 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-run\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-host\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-host\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.584872 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.584933 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.084915176 +0000 UTC m=+3.125509110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-konnectivity-ca\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.584993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzjf\" (UniqueName: \"kubernetes.io/projected/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-kube-api-access-6xzjf\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-conf\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-systemd\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-lib-modules\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-lib-modules\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-host-slash\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.585840 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-var-lib-kubelet\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.585531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-modprobe-d\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-iptables-alerter-script\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-kubernetes\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-serviceca\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-sysctl-conf\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-dbus\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-etc-systemd\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.586899 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c262271c-d309-4bed-9c9d-093d5335e24e-sys\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.587308 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.586992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-konnectivity-ca\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.587852 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.587825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-tmp\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.589351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.589324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5568d0d7-5e2c-45b5-94e3-2c412dfff0c8-agent-certs\") pod \"konnectivity-agent-l67cb\" (UID: \"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8\") " pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.589983 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.589936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c262271c-d309-4bed-9c9d-093d5335e24e-etc-tuned\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.601385 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.601328 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:03.601385 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.601356 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:03.601385 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.601372 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:03.601597 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:03.601435 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.101414622 +0000 UTC m=+3.142008581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:03.604908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.604879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfvg\" (UniqueName: \"kubernetes.io/projected/e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7-kube-api-access-rkfvg\") pod \"node-ca-bhsxg\" (UID: \"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7\") " pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.606364 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.606341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gsd\" (UniqueName: \"kubernetes.io/projected/dffacf5a-422a-42f5-a9c4-65f769ed48ab-kube-api-access-f6gsd\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:03.610943 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.610920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsx8\" (UniqueName: \"kubernetes.io/projected/c262271c-d309-4bed-9c9d-093d5335e24e-kube-api-access-2qsx8\") pod \"tuned-d4v2t\" (UID: \"c262271c-d309-4bed-9c9d-093d5335e24e\") " pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.615932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.615905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzjf\" (UniqueName: \"kubernetes.io/projected/8acf9bf0-7efc-4728-b4e3-2444b8caba7a-kube-api-access-6xzjf\") pod \"iptables-alerter-vz94h\" (UID: \"8acf9bf0-7efc-4728-b4e3-2444b8caba7a\") " pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.670541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.670502 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrbdq" Sep 16 13:43:03.679730 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.679703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:03.686733 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.686706 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d54b1b_882a_487d_b624_2840f40fb916.slice/crio-4aaf83ecba8ea4969f0cc6d1718858be5a9f9a05b3de95909af32a532182c790 WatchSource:0}: Error finding container 4aaf83ecba8ea4969f0cc6d1718858be5a9f9a05b3de95909af32a532182c790: Status 404 returned error can't find the container with id 4aaf83ecba8ea4969f0cc6d1718858be5a9f9a05b3de95909af32a532182c790 Sep 16 13:43:03.690271 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.689919 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" Sep 16 13:43:03.693400 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.693123 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272e92b2_262a_4d7d_b2af_38c8f9c2419d.slice/crio-924fdb8f8739f942ab313aa99d9e64f76885c5fffecf73e0348c83d6cdf3b4ca WatchSource:0}: Error finding container 924fdb8f8739f942ab313aa99d9e64f76885c5fffecf73e0348c83d6cdf3b4ca: Status 404 returned error can't find the container with id 924fdb8f8739f942ab313aa99d9e64f76885c5fffecf73e0348c83d6cdf3b4ca Sep 16 13:43:03.696670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.696648 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" Sep 16 13:43:03.703108 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.703081 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612e8a01_9d6f_4bf6_b499_b60d90f9bad4.slice/crio-98119fd23d66e78e4945f063d08da7846668be5291e525ea74841e43e09e24f3 WatchSource:0}: Error finding container 98119fd23d66e78e4945f063d08da7846668be5291e525ea74841e43e09e24f3: Status 404 returned error can't find the container with id 98119fd23d66e78e4945f063d08da7846668be5291e525ea74841e43e09e24f3 Sep 16 13:43:03.704498 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.704478 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:03.709745 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.709466 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adf8cef_fca8_4a55_be3f_5439845b971a.slice/crio-c940df134c20d7d5efe46057c2dcf93146ac58b616d94a562834b38a7745851b WatchSource:0}: Error finding container c940df134c20d7d5efe46057c2dcf93146ac58b616d94a562834b38a7745851b: Status 404 returned error can't find the container with id c940df134c20d7d5efe46057c2dcf93146ac58b616d94a562834b38a7745851b Sep 16 13:43:03.712302 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.711175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rr9k5" Sep 16 13:43:03.715349 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.715318 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:43:03.719223 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.719198 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bhsxg" Sep 16 13:43:03.726429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.726381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vz94h" Sep 16 13:43:03.731140 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:03.731114 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" Sep 16 13:43:03.736705 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.736664 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b9189c_5eb4_4da6_a0fa_6672a6eb69a7.slice/crio-a57e6bb21bd338cf5a670f4c3a674d9410f0a2d604455c5247339f30044c3135 WatchSource:0}: Error finding container a57e6bb21bd338cf5a670f4c3a674d9410f0a2d604455c5247339f30044c3135: Status 404 returned error can't find the container with id a57e6bb21bd338cf5a670f4c3a674d9410f0a2d604455c5247339f30044c3135 Sep 16 13:43:03.746460 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.746427 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acf9bf0_7efc_4728_b4e3_2444b8caba7a.slice/crio-2aecc1e0d2616feb5ff61d1ba2fd81335585fc60186fbb530cc09489f9d92fe8 WatchSource:0}: Error finding container 2aecc1e0d2616feb5ff61d1ba2fd81335585fc60186fbb530cc09489f9d92fe8: Status 404 returned error can't find the container with id 2aecc1e0d2616feb5ff61d1ba2fd81335585fc60186fbb530cc09489f9d92fe8 Sep 16 13:43:03.750085 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:03.750037 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc262271c_d309_4bed_9c9d_093d5335e24e.slice/crio-74024b2f9da4c6158903ccd2c61a353a5c13e6dcf162e89d4533b35bb4521fe9 WatchSource:0}: Error finding container 74024b2f9da4c6158903ccd2c61a353a5c13e6dcf162e89d4533b35bb4521fe9: Status 404 returned error can't find the container with id 74024b2f9da4c6158903ccd2c61a353a5c13e6dcf162e89d4533b35bb4521fe9 Sep 16 13:43:04.089000 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.088702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:04.089173 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.088888 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:04.089173 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.089092 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:05.08907066 +0000 UTC m=+4.129664608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:04.089173 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.089094 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:04.089173 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.089018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:04.089173 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.089149 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:05.08913351 +0000 UTC m=+4.129727460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:04.190304 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.190272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:04.190503 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.190479 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:04.190566 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.190510 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:04.190566 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.190523 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:04.190650 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:04.190589 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:05.19056951 +0000 UTC m=+4.231163444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:04.433579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.433472 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:38:03 +0000 UTC" deadline="2028-06-26 19:41:51.027026894 +0000 UTC" Sep 16 13:43:04.433579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.433516 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15581h58m46.593517017s" Sep 16 13:43:04.478386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.478347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" event={"ID":"c262271c-d309-4bed-9c9d-093d5335e24e","Type":"ContainerStarted","Data":"74024b2f9da4c6158903ccd2c61a353a5c13e6dcf162e89d4533b35bb4521fe9"} Sep 16 13:43:04.481053 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.481021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vz94h" event={"ID":"8acf9bf0-7efc-4728-b4e3-2444b8caba7a","Type":"ContainerStarted","Data":"2aecc1e0d2616feb5ff61d1ba2fd81335585fc60186fbb530cc09489f9d92fe8"} Sep 16 13:43:04.482785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.482758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rr9k5" event={"ID":"0d63c179-5550-4ee7-b927-bd95005a4d99","Type":"ContainerStarted","Data":"2ce65b51857da5392d57f927d24663ba3c7a17b2ebc69024f57667b557ab09e5"} Sep 16 13:43:04.484528 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.484501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerStarted","Data":"c940df134c20d7d5efe46057c2dcf93146ac58b616d94a562834b38a7745851b"} Sep 16 13:43:04.485939 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.485915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" event={"ID":"612e8a01-9d6f-4bf6-b499-b60d90f9bad4","Type":"ContainerStarted","Data":"98119fd23d66e78e4945f063d08da7846668be5291e525ea74841e43e09e24f3"} Sep 16 13:43:04.487675 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.487646 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"924fdb8f8739f942ab313aa99d9e64f76885c5fffecf73e0348c83d6cdf3b4ca"} Sep 16 13:43:04.489395 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.489371 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhsxg" event={"ID":"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7","Type":"ContainerStarted","Data":"a57e6bb21bd338cf5a670f4c3a674d9410f0a2d604455c5247339f30044c3135"} Sep 16 13:43:04.490995 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.490971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l67cb" event={"ID":"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8","Type":"ContainerStarted","Data":"7be5cf80eb7a4bde31e0467d74f6854a3fca03b40424f8e77e35f91ec993a1a3"} Sep 16 13:43:04.492957 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:04.492928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrbdq" event={"ID":"96d54b1b-882a-487d-b624-2840f40fb916","Type":"ContainerStarted","Data":"4aaf83ecba8ea4969f0cc6d1718858be5a9f9a05b3de95909af32a532182c790"} Sep 16 13:43:05.099441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.098457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:05.099441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.098533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:05.103080 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.103048 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:05.103215 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.103145 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.103124154 +0000 UTC m=+6.143718100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:05.103735 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.103712 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:05.103842 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.103787 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.10375723 +0000 UTC m=+6.144351166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:05.199318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.199277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:05.199494 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.199464 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:05.199559 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.199494 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:05.199559 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.199509 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:05.199660 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.199580 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.1995605 +0000 UTC m=+6.240154453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:05.434028 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.433923 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:38:03 +0000 UTC" deadline="2028-03-12 06:48:56.941190246 +0000 UTC" Sep 16 13:43:05.434028 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.433982 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13025h5m51.507211892s" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.467984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.467984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.468106 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:05.468118 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.468185 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:05.468348 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:05.468250 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:06.514657 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:06.514613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" event={"ID":"6e63221b9a751cb21bdf822925050ce0","Type":"ContainerStarted","Data":"c7557b49f163b7b72d06a6b980ac572ec4a6bf410abcad626ff7dcd620863188"} Sep 16 13:43:06.530222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:06.530162 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" event={"ID":"90413d152f265712b7c74d8e704e0cb6","Type":"ContainerStarted","Data":"f571ed597a559829794672f45c9d6847fe8b2d2f7bfee824cbec29c76903a6c1"} Sep 16 13:43:07.120097 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.120054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:07.120410 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.120120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:07.120410 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.120271 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:07.120410 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.120333 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:11.120314372 +0000 UTC m=+10.160908311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:07.120410 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.120271 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:07.120410 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.120397 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:11.120385835 +0000 UTC m=+10.160979784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:07.221160 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.220525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:07.221160 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.220707 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:07.221160 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.220726 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:07.221160 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.220738 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:07.221160 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.220799 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:11.220779588 +0000 UTC m=+10.261373523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:07.466585 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.465958 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:07.466585 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.466078 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:07.466585 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.466430 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:07.466585 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.466520 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:07.467163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:07.466994 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:07.467163 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:07.467099 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:09.465263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.465155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:09.465263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.465175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:09.465771 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:09.465301 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:09.465771 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:09.465365 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:09.465771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.465384 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:09.465771 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:09.465483 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:09.539182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.539089 2559 generic.go:356] "Generic (PLEG): container finished" podID="6e63221b9a751cb21bdf822925050ce0" containerID="c7557b49f163b7b72d06a6b980ac572ec4a6bf410abcad626ff7dcd620863188" exitCode=0 Sep 16 13:43:09.539182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.539138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" event={"ID":"6e63221b9a751cb21bdf822925050ce0","Type":"ContainerDied","Data":"c7557b49f163b7b72d06a6b980ac572ec4a6bf410abcad626ff7dcd620863188"} Sep 16 13:43:09.564029 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:09.563334 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-34.ec2.internal" podStartSLOduration=7.563313132 podStartE2EDuration="7.563313132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:06.552000101 +0000 UTC m=+5.592594057" watchObservedRunningTime="2026-09-16 13:43:09.563313132 +0000 UTC m=+8.603907088" Sep 16 13:43:11.149722 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.149673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:11.150120 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.149769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:11.150120 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.149786 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:11.150120 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.149848 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:19.149830459 +0000 UTC m=+18.190424417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:11.150120 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.149852 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:11.150120 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.149903 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:19.149892119 +0000 UTC m=+18.190486055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:11.250857 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.250130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:11.250857 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.250374 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:11.250857 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.250428 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:11.250857 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.250443 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:11.250857 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.250503 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:19.250485304 +0000 UTC m=+18.291079239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:11.465970 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.465906 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:11.465970 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.465901 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:11.465970 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:11.465913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:11.467167 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.466897 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:11.467167 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.467054 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:11.467167 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:11.467161 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:13.465410 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:13.465358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:13.465901 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:13.465370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:13.465901 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:13.465493 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:13.465901 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:13.465569 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:13.465901 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:13.465370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:13.465901 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:13.465676 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:15.465592 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:15.465555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:15.465592 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:15.465568 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:15.465592 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:15.465572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:15.466176 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:15.465715 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:15.466176 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:15.465777 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:15.466176 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:15.465902 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:17.465659 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:17.465610 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:17.465659 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:17.465660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:17.466131 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:17.465661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:17.466131 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:17.465735 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:17.466131 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:17.465825 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:17.466131 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:17.465924 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:19.215214 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.215172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:19.215693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.215230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:19.215693 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.215320 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:19.215693 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.215376 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:19.215693 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.215396 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.215375055 +0000 UTC m=+34.255968991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:19.215693 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.215423 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.215407774 +0000 UTC m=+34.256001721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:19.315918 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.315887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:19.316095 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.316048 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:19.316095 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.316071 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:19.316095 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.316081 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:19.316230 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.316128 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.316112692 +0000 UTC m=+34.356706625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:19.465698 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.465574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:19.465698 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.465577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:19.465941 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.465719 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:19.465941 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:19.465582 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:19.465941 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.465800 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:19.465941 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:19.465889 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:21.467006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:21.466970 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:21.467439 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:21.467068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:21.467439 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:21.467106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:21.467439 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:21.467110 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:21.467439 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:21.467181 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:21.467439 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:21.467262 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:22.566457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"6db617d2718e016fccfc03baac69c0e814601a5bacf4c0dd60ce9cad9fbe5a0d"} Sep 16 13:43:22.567012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566503 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"d06839c2404f08e7dfc44ea1ab91e21463742f6a17400f017a74f6f73d9eed21"} Sep 16 13:43:22.567012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566552 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"87af7b49aba594d68f27d79efe5cb2b5be40edcfe2609b80854d49839b9d9dc6"} Sep 16 13:43:22.567012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566589 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"ca0541ec26eb545f064686ebb62e59f80aafaf1f2f4f719e9199d9192fbbc4e4"} Sep 16 13:43:22.567012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566620 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"426f1ed65fb38ad4fdf3bba9b53d2c4ec0e53a311fd6f495f620c1e6c217c905"} Sep 16 13:43:22.567012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.566652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"ccb6324c147dbf391aad1e7bffe23d08c83d443122112a760ae1634df23da42c"} Sep 16 13:43:22.567904 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.567864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" event={"ID":"6e63221b9a751cb21bdf822925050ce0","Type":"ContainerStarted","Data":"9dd4073a956dedb9b2edc8958efa43005c627e3e0e2718d4d636ecc439bb5a29"} Sep 16 13:43:22.569183 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.569164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bhsxg" event={"ID":"e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7","Type":"ContainerStarted","Data":"a4f705d8d0d24085b382be5fa7d15940df58b4961cd4e87d69c9cbe657b76025"} Sep 16 13:43:22.570570 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.570552 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l67cb" event={"ID":"5568d0d7-5e2c-45b5-94e3-2c412dfff0c8","Type":"ContainerStarted","Data":"7c6bee0445936ea7576a7709fbacf677bda419f63fc1675dc20890c2a5a656e3"} Sep 16 13:43:22.571918 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.571898 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrbdq" event={"ID":"96d54b1b-882a-487d-b624-2840f40fb916","Type":"ContainerStarted","Data":"e67f43b555424161b3982c99cb5571430373907f8bc72ce5292aafdf53937f3f"} Sep 16 13:43:22.573303 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.573282 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" event={"ID":"c262271c-d309-4bed-9c9d-093d5335e24e","Type":"ContainerStarted","Data":"413dbc7728c5a78c17cb3ee6fb57be3363eea84de3e467f70260ae48f0097440"} Sep 16 13:43:22.574568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.574548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rr9k5" event={"ID":"0d63c179-5550-4ee7-b927-bd95005a4d99","Type":"ContainerStarted","Data":"9149773c3c6b40331b3ebac5698dacb0870914ef055e431efc1e337579e630c5"} Sep 16 13:43:22.575970 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.575951 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="cbe2d7307f42e4b5a5a510aef2b8e5c437c29ac73e0316e0eb23a2cf583642cd" exitCode=0 Sep 16 13:43:22.576042 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.575973 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"cbe2d7307f42e4b5a5a510aef2b8e5c437c29ac73e0316e0eb23a2cf583642cd"} Sep 16 13:43:22.577483 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.577465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" event={"ID":"612e8a01-9d6f-4bf6-b499-b60d90f9bad4","Type":"ContainerStarted","Data":"5e033043df3ad24a1a390339f2ad0db0a8cc6da741299914ec3d397dc8624c55"} Sep 16 13:43:22.586694 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.586654 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-34.ec2.internal" podStartSLOduration=20.586621318 podStartE2EDuration="20.586621318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.586594809 +0000 UTC m=+21.627188767" watchObservedRunningTime="2026-09-16 13:43:22.586621318 +0000 UTC m=+21.627215270" Sep 16 13:43:22.603403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.603361 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l67cb" podStartSLOduration=7.764050812 podStartE2EDuration="21.603347317s" totalImagesPullingTime="13.839296505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.602955517 +0000 UTC m=+21.643549485" watchObservedRunningTime="2026-09-16 13:43:22.603347317 +0000 UTC m=+21.643941299" Sep 16 13:43:22.620236 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.620195 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zrbdq" podStartSLOduration=3.590230095 podStartE2EDuration="21.620182162s" totalImagesPullingTime="18.029952067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.620136767 +0000 UTC m=+21.660730733" watchObservedRunningTime="2026-09-16 13:43:22.620182162 +0000 UTC m=+21.660776116" Sep 16 13:43:22.663960 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.663915 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bhsxg" podStartSLOduration=3.664795417 podStartE2EDuration="21.663900069s" totalImagesPullingTime="17.999104652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.663767365 +0000 UTC m=+21.704361314" watchObservedRunningTime="2026-09-16 13:43:22.663900069 +0000 UTC m=+21.704494026" Sep 16 13:43:22.691182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.691135 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d4v2t" podStartSLOduration=3.701120371 podStartE2EDuration="21.691118667s" totalImagesPullingTime="17.989998296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.690892003 +0000 UTC m=+21.731485972" watchObservedRunningTime="2026-09-16 13:43:22.691118667 +0000 UTC m=+21.731712621" Sep 16 13:43:22.712235 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.712188 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rr9k5" podStartSLOduration=3.579755832 podStartE2EDuration="21.712174317s" totalImagesPullingTime="18.132418485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.712154086 +0000 UTC m=+21.752748041" watchObservedRunningTime="2026-09-16 13:43:22.712174317 +0000 UTC m=+21.752768271" Sep 16 13:43:22.797919 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.797859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:22.879743 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.879652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:22.966402 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:22.966368 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:23.195468 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.195439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:23.264968 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.264949 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 13:43:23.429234 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.429073 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:43:23.264965278Z","UUID":"91441f35-917a-425f-a75b-b9b4104acf49","Handler":null,"Name":"","Endpoint":""} Sep 16 13:43:23.431163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.431131 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 13:43:23.431163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.431162 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 13:43:23.465860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.465825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:23.466047 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.465825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:23.466047 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:23.465979 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:23.466047 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:23.466035 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:23.466183 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.465830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:23.466183 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:23.466141 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:23.581040 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.581001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vz94h" event={"ID":"8acf9bf0-7efc-4728-b4e3-2444b8caba7a","Type":"ContainerStarted","Data":"5450f87f246ce6d06c27a2424ab4150dcf6d0ed184794e445ba2607579761fdf"} Sep 16 13:43:23.583240 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.583210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" event={"ID":"612e8a01-9d6f-4bf6-b499-b60d90f9bad4","Type":"ContainerStarted","Data":"ec3389ee42ea47dfe6f4cc224b26e7d71ca45c99824bff0bf84748c67bbbdc2e"} Sep 16 13:43:23.598945 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.598889 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vz94h" podStartSLOduration=4.608233821 podStartE2EDuration="22.598851468s" totalImagesPullingTime="17.990617647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:23.598043241 +0000 UTC m=+22.638637195" watchObservedRunningTime="2026-09-16 13:43:23.598851468 +0000 UTC m=+22.639445425" Sep 16 13:43:23.733835 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.733795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:23.796124 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:23.796098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:24.588755 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:24.588401 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" event={"ID":"612e8a01-9d6f-4bf6-b499-b60d90f9bad4","Type":"ContainerStarted","Data":"8e02f8a0074de2e961e95540b14d1b3868ddacdf6a476f9092b5ec4edd1ea2d9"} Sep 16 13:43:24.592849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:24.592809 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"df76f083a54799ffa344578936112271e7e83ecae371acc08f92c5d7beda04d3"} Sep 16 13:43:24.618171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:24.618123 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsw8w" podStartSLOduration=3.399750961 podStartE2EDuration="23.618104054s" totalImagesPullingTime="20.218353093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:24.617483907 +0000 UTC m=+23.658077861" watchObservedRunningTime="2026-09-16 13:43:24.618104054 +0000 UTC m=+23.658698012" Sep 16 13:43:25.259633 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.259599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:25.360308 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.360266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:25.466084 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.466054 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:25.466258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.466065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:25.466258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.466181 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:25.466258 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:25.466174 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:25.466412 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:25.466306 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:25.466468 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:25.466396 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:25.895879 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.895826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:25.958779 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:25.958747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:26.844636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:26.844547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:26.845270 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:26.845240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:27.468747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.468510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:27.468747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.468525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:27.468747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.468544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:27.468747 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:27.468621 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:27.470096 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:27.468757 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:27.470096 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:27.468822 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:27.603744 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.603645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" event={"ID":"272e92b2-262a-4d7d-b2af-38c8f9c2419d","Type":"ContainerStarted","Data":"cf4598d5d452fac791d9531d9b30357ade68612ec9193e85c7d68eda65e8bd75"} Sep 16 13:43:27.603955 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.603930 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:27.604261 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.604236 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:27.604382 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.604309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:27.604382 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.604358 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:27.604842 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.604793 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l67cb" Sep 16 13:43:27.619347 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.619155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:27.619623 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.619608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:27.649413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.649363 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" podStartSLOduration=8.591617424 podStartE2EDuration="26.649348931s" totalImagesPullingTime="18.057731507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:27.634960738 +0000 UTC m=+26.675554693" watchObservedRunningTime="2026-09-16 13:43:27.649348931 +0000 UTC m=+26.689942878" Sep 16 13:43:27.747542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:27.747507 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:28.607644 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:28.607607 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="563326813dd1c034a46966c64277e5dacc515a7b7f70fe31473e8c2249f25075" exitCode=0 Sep 16 13:43:28.608179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:28.607695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"563326813dd1c034a46966c64277e5dacc515a7b7f70fe31473e8c2249f25075"} Sep 16 13:43:29.468584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.468553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:29.468584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.468581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:29.468747 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.468668 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:29.468747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.468722 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:29.468825 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.468800 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:29.468886 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.468857 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:29.507222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.507191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:29.542550 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.542470 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6d6sc"] Sep 16 13:43:29.548773 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.548746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8stjr"] Sep 16 13:43:29.567100 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.567075 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cwjfb"] Sep 16 13:43:29.611934 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.611905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:29.612547 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.612197 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="b4046f8cec9f5fecab1c3d650a25f2ac552f760ef0eb77087a586b49f353f957" exitCode=0 Sep 16 13:43:29.612547 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.612275 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:29.612547 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.612302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"b4046f8cec9f5fecab1c3d650a25f2ac552f760ef0eb77087a586b49f353f957"} Sep 16 13:43:29.612547 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.612369 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:29.612547 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.612457 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:29.612723 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.612641 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:29.612893 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:29.612878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:29.613335 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:29.613303 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:30.495688 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:30.495520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:30.617503 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:30.617477 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="87c720f74eed677d08c9a120e7c1743116aaf0f0012f9ea26c87c83515df7ecd" exitCode=0 Sep 16 13:43:30.617798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:30.617534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"87c720f74eed677d08c9a120e7c1743116aaf0f0012f9ea26c87c83515df7ecd"} Sep 16 13:43:30.889556 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:30.889481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:31.077437 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:31.077403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:31.469166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:31.469093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:31.469166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:31.469141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:31.469357 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:31.469103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:31.469357 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:31.469237 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:31.469357 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:31.469315 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:31.469574 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:31.469371 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:31.983944 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:31.983910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.099860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.099821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.208227 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.208193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.288394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.288292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.517860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.517825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.608345 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.608253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.765112 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.765083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.914157 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:32.914071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.012350 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:33.012314 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.465563 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:33.465531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:33.465731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:33.465531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:33.465731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:33.465676 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:33.465835 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:33.465670 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwjfb" podUID="dffacf5a-422a-42f5-a9c4-65f769ed48ab" Sep 16 13:43:33.465835 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:33.465783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6d6sc" podUID="1a148477-2b87-4b48-bde5-b0f21b9b2ac9" Sep 16 13:43:33.465943 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:33.465913 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8stjr" podUID="c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0" Sep 16 13:43:33.582523 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:33.582497 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrbdq_96d54b1b-882a-487d-b624-2840f40fb916/dns-node-resolver/0.log" Sep 16 13:43:34.181164 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.181132 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bhsxg_e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7/node-ca/0.log" Sep 16 13:43:34.601006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.600969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:34.707736 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.707687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:34.756250 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.756227 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-34.ec2.internal" event="NodeReady" Sep 16 13:43:34.756413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.756360 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 13:43:34.823622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.823592 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lvlqk"] Sep 16 13:43:34.838064 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.837997 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pgzsc"] Sep 16 13:43:34.838196 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.838078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:34.841486 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.841462 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 13:43:34.842113 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.842081 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 13:43:34.842113 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.842112 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kqx7z\"" Sep 16 13:43:34.854264 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.854210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvlqk"] Sep 16 13:43:34.854376 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.854354 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.857144 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857119 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 13:43:34.857144 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857129 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dks\"" Sep 16 13:43:34.857399 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857382 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 13:43:34.857505 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857465 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgzsc"] Sep 16 13:43:34.857553 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857528 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 13:43:34.857661 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.857642 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 13:43:34.865625 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.865602 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xn5kz"] Sep 16 13:43:34.884942 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.884916 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:34.887335 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.887313 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 13:43:34.887443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.887367 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 13:43:34.887443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.887431 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4dzm\"" Sep 16 13:43:34.887631 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.887619 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 13:43:34.892135 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.892116 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xn5kz"] Sep 16 13:43:34.931483 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xks\" (UniqueName: \"kubernetes.io/projected/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-api-access-d2xks\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.931658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-crio-socket\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.931658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-data-volume\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.931658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.931658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:34.931658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:34.931850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931667 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c830c48-2260-48df-84b1-b919795bc7c0-cert\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:34.931850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9vk\" (UniqueName: \"kubernetes.io/projected/0c830c48-2260-48df-84b1-b919795bc7c0-kube-api-access-nm9vk\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:34.931850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931747 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-config-volume\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:34.931850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgvz\" (UniqueName: \"kubernetes.io/projected/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-kube-api-access-7rgvz\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:34.931850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:34.931808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-tmp-dir\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.006313 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.006278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:35.032729 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9vk\" (UniqueName: \"kubernetes.io/projected/0c830c48-2260-48df-84b1-b919795bc7c0-kube-api-access-nm9vk\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:35.032729 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-config-volume\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.032952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgvz\" (UniqueName: \"kubernetes.io/projected/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-kube-api-access-7rgvz\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.032952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-tmp-dir\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.032952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xks\" (UniqueName: \"kubernetes.io/projected/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-api-access-d2xks\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.032978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-crio-socket\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-data-volume\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-tmp-dir\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033126 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033194 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.533177076 +0000 UTC m=+34.573771010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : configmap "kube-rbac-proxy" not found Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033219 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.033238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-crio-socket\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c830c48-2260-48df-84b1-b919795bc7c0-cert\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033278 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.53326036 +0000 UTC m=+34.573854293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033286 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.033327 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls podName:17ccbdc0-e1e0-4459-9ea8-e6b04119d22d nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.533316154 +0000 UTC m=+34.573910092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls") pod "dns-default-lvlqk" (UID: "17ccbdc0-e1e0-4459-9ea8-e6b04119d22d") : secret "dns-default-metrics-tls" not found Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-config-volume\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.033544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.033394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-data-volume\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.037132 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.037108 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c830c48-2260-48df-84b1-b919795bc7c0-cert\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:35.044724 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.044699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9vk\" (UniqueName: \"kubernetes.io/projected/0c830c48-2260-48df-84b1-b919795bc7c0-kube-api-access-nm9vk\") pod \"ingress-canary-xn5kz\" (UID: \"0c830c48-2260-48df-84b1-b919795bc7c0\") " pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:35.046150 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.046123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xks\" (UniqueName: \"kubernetes.io/projected/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-api-access-d2xks\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.046150 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.046141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgvz\" (UniqueName: \"kubernetes.io/projected/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-kube-api-access-7rgvz\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.107365 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.107290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:35.193029 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.193001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xn5kz" Sep 16 13:43:35.235332 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.235171 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:35.235461 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.235281 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:35.235461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.235407 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:35.235461 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.235446 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs podName:dffacf5a-422a-42f5-a9c4-65f769ed48ab nodeName:}" failed. No retries permitted until 2026-09-16 13:44:07.23543076 +0000 UTC m=+66.276024709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs") pod "network-metrics-daemon-cwjfb" (UID: "dffacf5a-422a-42f5-a9c4-65f769ed48ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:35.235630 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.235505 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:35.235630 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.235547 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret podName:c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:07.235537425 +0000 UTC m=+66.276131358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret") pod "global-pull-secret-syncer-8stjr" (UID: "c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:35.335914 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.335883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:35.336090 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.336073 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:35.336158 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.336101 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:35.336158 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.336114 2559 projected.go:196] Error preparing data for projected volume kube-api-access-8c9hf for pod openshift-network-diagnostics/network-check-target-6d6sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:35.336236 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.336179 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf podName:1a148477-2b87-4b48-bde5-b0f21b9b2ac9 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:07.336160367 +0000 UTC m=+66.376754302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c9hf" (UniqueName: "kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf") pod "network-check-target-6d6sc" (UID: "1a148477-2b87-4b48-bde5-b0f21b9b2ac9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:35.465907 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.465856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:43:35.465907 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.465891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:43:35.466149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.465860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:43:35.469397 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.469374 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:43:35.469526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.469415 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:43:35.469526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.469418 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:43:35.469723 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.469708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\"" Sep 16 13:43:35.470023 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.470006 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:43:35.470238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.470224 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\"" Sep 16 13:43:35.524529 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.524499 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xn5kz"] Sep 16 13:43:35.527990 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:35.527957 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c830c48_2260_48df_84b1_b919795bc7c0.slice/crio-b357fe02c1abb453c36f6711355af300c9ef0c668ca954e393074b2e16897166 WatchSource:0}: Error finding container b357fe02c1abb453c36f6711355af300c9ef0c668ca954e393074b2e16897166: Status 404 returned error can't find the container with id b357fe02c1abb453c36f6711355af300c9ef0c668ca954e393074b2e16897166 Sep 16 13:43:35.536776 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.536752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.536860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.536824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:35.536860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.536854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:35.536948 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.536883 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:35.536948 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.536946 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.53692671 +0000 UTC m=+35.577520649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : configmap "kube-rbac-proxy" not found Sep 16 13:43:35.537031 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.537011 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:35.537074 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.537049 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls podName:17ccbdc0-e1e0-4459-9ea8-e6b04119d22d nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.537040449 +0000 UTC m=+35.577634382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls") pod "dns-default-lvlqk" (UID: "17ccbdc0-e1e0-4459-9ea8-e6b04119d22d") : secret "dns-default-metrics-tls" not found Sep 16 13:43:35.537074 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.537054 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.537162 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:35.537114 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.53709922 +0000 UTC m=+35.577693173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.632354 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.632322 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="18c46dffc8391b880a52b46c6033093e4317e1183aeab360d385219fd1ceb9af" exitCode=0 Sep 16 13:43:35.632544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.632390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"18c46dffc8391b880a52b46c6033093e4317e1183aeab360d385219fd1ceb9af"} Sep 16 13:43:35.633760 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.633727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xn5kz" event={"ID":"0c830c48-2260-48df-84b1-b919795bc7c0","Type":"ContainerStarted","Data":"b357fe02c1abb453c36f6711355af300c9ef0c668ca954e393074b2e16897166"} Sep 16 13:43:35.759702 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:35.759670 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:36.130605 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.130572 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:36.225882 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.225834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:36.330595 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.330513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:36.546917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.546854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:36.546917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.546917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:36.547179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.547013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:36.547179 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:36.547050 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:36.547179 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:36.547096 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:36.547179 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:36.547129 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:38.547105625 +0000 UTC m=+37.587699569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:36.547179 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:36.547154 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls podName:17ccbdc0-e1e0-4459-9ea8-e6b04119d22d nodeName:}" failed. No retries permitted until 2026-09-16 13:43:38.547136999 +0000 UTC m=+37.587730938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls") pod "dns-default-lvlqk" (UID: "17ccbdc0-e1e0-4459-9ea8-e6b04119d22d") : secret "dns-default-metrics-tls" not found Sep 16 13:43:36.547602 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.547581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:36.640198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.640124 2559 generic.go:356] "Generic (PLEG): container finished" podID="6adf8cef-fca8-4a55-be3f-5439845b971a" containerID="169925155f05834f5f573e2b1efc4a572d610deeef69b4ef1fc2b7f3c3f2a6ba" exitCode=0 Sep 16 13:43:36.640335 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:36.640188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerDied","Data":"169925155f05834f5f573e2b1efc4a572d610deeef69b4ef1fc2b7f3c3f2a6ba"} Sep 16 13:43:37.647511 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:37.647467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" event={"ID":"6adf8cef-fca8-4a55-be3f-5439845b971a","Type":"ContainerStarted","Data":"e22c49be84404d0c3f623a833a99296c618ba56feb05c3a7bff30b4943862fd5"} Sep 16 13:43:37.678979 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:37.678688 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5w2tn" podStartSLOduration=7.644794919 podStartE2EDuration="36.678663961s" totalImagesPullingTime="29.033869042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:37.678141951 +0000 UTC m=+36.718735907" watchObservedRunningTime="2026-09-16 13:43:37.678663961 +0000 UTC m=+36.719257918" Sep 16 13:43:38.480853 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.480820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:38.565705 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.565666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:38.565705 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.565705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:38.565918 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:38.565814 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:38.565918 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:38.565818 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:38.565918 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:38.565889 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls podName:17ccbdc0-e1e0-4459-9ea8-e6b04119d22d nodeName:}" failed. No retries permitted until 2026-09-16 13:43:42.565856387 +0000 UTC m=+41.606450327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls") pod "dns-default-lvlqk" (UID: "17ccbdc0-e1e0-4459-9ea8-e6b04119d22d") : secret "dns-default-metrics-tls" not found Sep 16 13:43:38.565918 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:38.565904 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:42.565898004 +0000 UTC m=+41.606491936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:38.651839 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.651798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xn5kz" event={"ID":"0c830c48-2260-48df-84b1-b919795bc7c0","Type":"ContainerStarted","Data":"ed06e05a1378c5846be3e8e298be3c0c335149395f8d6d7bc6eb08575758c33f"} Sep 16 13:43:38.675884 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.675822 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xn5kz" podStartSLOduration=2.126249476 podStartE2EDuration="4.675803616s" totalImagesPullingTime="2.54955414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:38.674939777 +0000 UTC m=+37.715533731" watchObservedRunningTime="2026-09-16 13:43:38.675803616 +0000 UTC m=+37.716397568" Sep 16 13:43:38.688327 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.688297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:38.819693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.819591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:38.937520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:38.937478 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:39.037516 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:39.037479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:39.268466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:39.268434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:39.369678 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:39.369640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:39.471110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:39.471081 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:39.581092 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:39.581013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:40.268752 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:40.268700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:40.571733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:40.571609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:40.991182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:40.991153 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:41.185313 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:41.185283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:41.351437 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:41.351346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:41.676960 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:41.676846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:41.776139 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:41.776090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:41.851476 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:41.851425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:42.157251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.157196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:42.597907 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.597847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:42.597907 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.597910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:42.598358 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:42.598005 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:42.598358 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:43:42.598070 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls podName:c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae nodeName:}" failed. No retries permitted until 2026-09-16 13:43:50.598053924 +0000 UTC m=+49.638647857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgzsc" (UID: "c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:42.601595 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.601559 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ccbdc0-e1e0-4459-9ea8-e6b04119d22d-metrics-tls\") pod \"dns-default-lvlqk\" (UID: \"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d\") " pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:42.648546 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.648506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:42.890533 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.890419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:42.975860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.975827 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvlqk"] Sep 16 13:43:42.979234 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:42.979207 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ccbdc0_e1e0_4459_9ea8_e6b04119d22d.slice/crio-655524bc274e0a72486656efe05a5f28d18f229523841a7b444ab486d75a2429 WatchSource:0}: Error finding container 655524bc274e0a72486656efe05a5f28d18f229523841a7b444ab486d75a2429: Status 404 returned error can't find the container with id 655524bc274e0a72486656efe05a5f28d18f229523841a7b444ab486d75a2429 Sep 16 13:43:42.983773 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:42.983737 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:43.390997 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.390959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:43.555228 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.555199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:43.634432 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.634404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:43.663904 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.663805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvlqk" event={"ID":"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d","Type":"ContainerStarted","Data":"655524bc274e0a72486656efe05a5f28d18f229523841a7b444ab486d75a2429"} Sep 16 13:43:43.752083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.752050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:43.968085 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:43.968050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:44.544658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:44.544623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:44.682383 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:44.682346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:44.780104 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:44.780061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:44.942825 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:44.942739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:45.562410 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:45.562378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:45.871347 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:45.871309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:45.964839 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:45.964805 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:46.359383 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.359349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:46.672522 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.672436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvlqk" event={"ID":"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d","Type":"ContainerStarted","Data":"7c8c6c81fedd03723daf238c08ebd24fcab43324de49a07d0c9ac435642350bd"} Sep 16 13:43:46.672714 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.672526 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvlqk" event={"ID":"17ccbdc0-e1e0-4459-9ea8-e6b04119d22d","Type":"ContainerStarted","Data":"a6e24400a329f46062fab676ef71e06e859fd3b34999c9e6c271930593b485f2"} Sep 16 13:43:46.672714 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.672570 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:46.697080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.697020 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lvlqk" podStartSLOduration=10.088933269 podStartE2EDuration="12.697004023s" totalImagesPullingTime="2.608070754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:46.695576928 +0000 UTC m=+45.736170885" watchObservedRunningTime="2026-09-16 13:43:46.697004023 +0000 UTC m=+45.737597978" Sep 16 13:43:46.758911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.758879 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:46.993765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:46.993719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:47.375917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:47.375820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.182096 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:48.182061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.409526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:48.409494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.590117 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:48.590088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.724379 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:48.724345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.847132 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:48.847055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:49.195094 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:49.195064 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:50.660260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:50.660226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:50.662459 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:50.662432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgzsc\" (UID: \"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae\") " pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:50.763358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:50.763325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgzsc" Sep 16 13:43:51.069190 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:51.069157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgzsc"] Sep 16 13:43:51.072917 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:51.072891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43e15af_2fdb_4dc5_a0dc_c9fa5f8f1eae.slice/crio-540dc68f2177152f7bc6428a4d75244c7a63a3de3c8128f9a85f9eab32c81c6b WatchSource:0}: Error finding container 540dc68f2177152f7bc6428a4d75244c7a63a3de3c8128f9a85f9eab32c81c6b: Status 404 returned error can't find the container with id 540dc68f2177152f7bc6428a4d75244c7a63a3de3c8128f9a85f9eab32c81c6b Sep 16 13:43:51.686321 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:51.686283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgzsc" event={"ID":"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae","Type":"ContainerStarted","Data":"ca4769891208ea5dbb0b6be6e14a78edcc329db2c01f303a4a1536b2c547a037"} Sep 16 13:43:51.686737 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:51.686350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgzsc" event={"ID":"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae","Type":"ContainerStarted","Data":"540dc68f2177152f7bc6428a4d75244c7a63a3de3c8128f9a85f9eab32c81c6b"} Sep 16 13:43:52.690030 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:52.689994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgzsc" event={"ID":"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae","Type":"ContainerStarted","Data":"8d77bb1b5444a00621de06f4a995d21a54c275dc4023004c9a6f8c355faa8464"} Sep 16 13:43:53.695638 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:53.695602 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgzsc" event={"ID":"c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae","Type":"ContainerStarted","Data":"429c17bf402d81145f04050da6dc352dcfe7624d739ebadc3156e5e8945ee7c6"} Sep 16 13:43:53.723633 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:53.723582 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pgzsc" podStartSLOduration=17.411690116 podStartE2EDuration="19.723566417s" totalImagesPullingTime="2.311876301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:53.723064828 +0000 UTC m=+52.763658784" watchObservedRunningTime="2026-09-16 13:43:53.723566417 +0000 UTC m=+52.764160372" Sep 16 13:43:54.125802 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:54.125771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:54.240674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:54.240643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:56.677248 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.677211 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lvlqk" Sep 16 13:43:56.947062 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.947034 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw"] Sep 16 13:43:56.948762 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.948745 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:56.954791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.954763 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f"] Sep 16 13:43:56.956778 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.956758 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk"] Sep 16 13:43:56.956881 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.956838 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 13:43:56.956942 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.956887 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:56.957016 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.956997 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w985c\"" Sep 16 13:43:56.958540 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.958522 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:56.960713 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.960699 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 13:43:56.968395 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.968374 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 13:43:56.974165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.974145 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 13:43:56.974345 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.974318 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 13:43:56.976695 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.976672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw"] Sep 16 13:43:56.977629 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.977597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 13:43:56.979176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.979157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f"] Sep 16 13:43:56.986388 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:56.986364 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 13:43:57.005618 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.005569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk"] Sep 16 13:43:57.005963 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.005945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dc9qm\"" Sep 16 13:43:57.111942 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.111902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085ba3a6-9ad8-447e-8995-b7241b06e1bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.111942 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.111948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085ba3a6-9ad8-447e-8995-b7241b06e1bd-tmp\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.112152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.111969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnvd\" (UniqueName: \"kubernetes.io/projected/085ba3a6-9ad8-447e-8995-b7241b06e1bd-kube-api-access-9fnvd\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.112152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.112025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.112152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.112098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.112152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.112125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.112152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.112142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndxs\" (UniqueName: \"kubernetes.io/projected/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-kube-api-access-kndxs\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.167468 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.167433 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b5f48df84-4rfvr"] Sep 16 13:43:57.169224 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.169207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.173684 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.173656 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4xvt\"" Sep 16 13:43:57.173838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.173660 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 13:43:57.173838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.173708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 13:43:57.173838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.173809 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 13:43:57.178623 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.178481 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 13:43:57.191648 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.191612 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5f48df84-4rfvr"] Sep 16 13:43:57.192198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.192160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.213267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085ba3a6-9ad8-447e-8995-b7241b06e1bd-tmp\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.213267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnvd\" (UniqueName: \"kubernetes.io/projected/085ba3a6-9ad8-447e-8995-b7241b06e1bd-kube-api-access-9fnvd\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.213267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.213536 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.213536 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.213536 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndxs\" (UniqueName: \"kubernetes.io/projected/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-kube-api-access-kndxs\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.213536 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085ba3a6-9ad8-447e-8995-b7241b06e1bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.213746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.213707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085ba3a6-9ad8-447e-8995-b7241b06e1bd-tmp\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.214099 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.214077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.216008 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.215982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.216088 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.216048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085ba3a6-9ad8-447e-8995-b7241b06e1bd-klusterlet-config\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.216132 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.216078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-w5jsw\" (UID: \"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.233626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.233593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnvd\" (UniqueName: \"kubernetes.io/projected/085ba3a6-9ad8-447e-8995-b7241b06e1bd-kube-api-access-9fnvd\") pod \"klusterlet-addon-workmgr-7777b69b8d-wv8xk\" (UID: \"085ba3a6-9ad8-447e-8995-b7241b06e1bd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.243657 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.243625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndxs\" (UniqueName: \"kubernetes.io/projected/924fa136-68e1-4ccb-8fb5-f8d4e677d8b0-kube-api-access-kndxs\") pod \"managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f\" (UID: \"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.258073 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.258023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" Sep 16 13:43:57.274763 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.274733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" Sep 16 13:43:57.280774 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.280743 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:43:57.313844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.313805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-certificates\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.313981 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.313926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-tls\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314042 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.313986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-ca-trust-extracted\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314042 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.314012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-bound-sa-token\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.314044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-image-registry-private-configuration\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.314073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-trusted-ca\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.314108 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-installation-pull-secrets\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.314129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7wb\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-kube-api-access-2b7wb\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.344966 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.344758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.415063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-certificates\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-tls\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-ca-trust-extracted\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-bound-sa-token\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-image-registry-private-configuration\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-trusted-ca\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-installation-pull-secrets\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415263 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7wb\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-kube-api-access-2b7wb\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.415773 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.415746 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-ca-trust-extracted\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.416026 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.416001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-certificates\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.416476 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.416454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-trusted-ca\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.417640 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.417622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-image-registry-private-configuration\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.417796 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.417771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-installation-pull-secrets\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.417881 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.417857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-registry-tls\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.428138 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.428102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-bound-sa-token\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.432351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.432329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7wb\" (UniqueName: \"kubernetes.io/projected/ab4dbb1d-5d1a-4836-980f-06e0f3be548a-kube-api-access-2b7wb\") pod \"image-registry-7b5f48df84-4rfvr\" (UID: \"ab4dbb1d-5d1a-4836-980f-06e0f3be548a\") " pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.481758 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.481668 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:57.525611 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.525554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.552337 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.552305 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw"] Sep 16 13:43:57.556477 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:57.556450 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b204615_e64d_4ccb_9ae0_e7c8bc4a95d3.slice/crio-48a0bc11b7014d343204932f41c438e35d9bfa9a83e33ba77dbb73663d073ca7 WatchSource:0}: Error finding container 48a0bc11b7014d343204932f41c438e35d9bfa9a83e33ba77dbb73663d073ca7: Status 404 returned error can't find the container with id 48a0bc11b7014d343204932f41c438e35d9bfa9a83e33ba77dbb73663d073ca7 Sep 16 13:43:57.572328 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.572299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f"] Sep 16 13:43:57.575559 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.575441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk"] Sep 16 13:43:57.575810 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:57.575789 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod924fa136_68e1_4ccb_8fb5_f8d4e677d8b0.slice/crio-23410423633b928ac5d0ea32739232a00b767278a180d9afa50853514334ab3e WatchSource:0}: Error finding container 23410423633b928ac5d0ea32739232a00b767278a180d9afa50853514334ab3e: Status 404 returned error can't find the container with id 23410423633b928ac5d0ea32739232a00b767278a180d9afa50853514334ab3e Sep 16 13:43:57.578347 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:57.578317 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085ba3a6_9ad8_447e_8995_b7241b06e1bd.slice/crio-85e4ffd2b2028cac7c530d83959105e7be3a8b28590e86c1d0498efc2560c892 WatchSource:0}: Error finding container 85e4ffd2b2028cac7c530d83959105e7be3a8b28590e86c1d0498efc2560c892: Status 404 returned error can't find the container with id 85e4ffd2b2028cac7c530d83959105e7be3a8b28590e86c1d0498efc2560c892 Sep 16 13:43:57.641374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.641338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.711237 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.711194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" event={"ID":"085ba3a6-9ad8-447e-8995-b7241b06e1bd","Type":"ContainerStarted","Data":"85e4ffd2b2028cac7c530d83959105e7be3a8b28590e86c1d0498efc2560c892"} Sep 16 13:43:57.712371 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.712352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" event={"ID":"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0","Type":"ContainerStarted","Data":"23410423633b928ac5d0ea32739232a00b767278a180d9afa50853514334ab3e"} Sep 16 13:43:57.713357 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.713336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" event={"ID":"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3","Type":"ContainerStarted","Data":"48a0bc11b7014d343204932f41c438e35d9bfa9a83e33ba77dbb73663d073ca7"} Sep 16 13:43:57.761820 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.761740 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5f48df84-4rfvr"] Sep 16 13:43:57.764097 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:57.764066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.765271 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:43:57.765250 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4dbb1d_5d1a_4836_980f_06e0f3be548a.slice/crio-29e0396be8d9c845a77e0b176763ff53e572853356e76b9bf54784b30c3cbe2c WatchSource:0}: Error finding container 29e0396be8d9c845a77e0b176763ff53e572853356e76b9bf54784b30c3cbe2c: Status 404 returned error can't find the container with id 29e0396be8d9c845a77e0b176763ff53e572853356e76b9bf54784b30c3cbe2c Sep 16 13:43:58.347544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.347511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.451761 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.451697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.692710 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.692602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.722777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.722245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" event={"ID":"ab4dbb1d-5d1a-4836-980f-06e0f3be548a","Type":"ContainerStarted","Data":"a22ec2a9ff611d87af4905e74a9b7be372f247a469213816489263a17e66ece5"} Sep 16 13:43:58.722777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.722309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" event={"ID":"ab4dbb1d-5d1a-4836-980f-06e0f3be548a","Type":"ContainerStarted","Data":"29e0396be8d9c845a77e0b176763ff53e572853356e76b9bf54784b30c3cbe2c"} Sep 16 13:43:58.724106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.723668 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:43:58.765654 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.765587 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" podStartSLOduration=1.765562755 podStartE2EDuration="1.765562755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:58.764757634 +0000 UTC m=+57.805351590" watchObservedRunningTime="2026-09-16 13:43:58.765562755 +0000 UTC m=+57.806156791" Sep 16 13:43:58.811851 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.811806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.974045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:58.973949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.565294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.565263 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-rbxrh"] Sep 16 13:43:59.565562 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.565526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.588897 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.588828 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rbxrh"] Sep 16 13:43:59.589286 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.589224 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:43:59.594401 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.594373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 13:43:59.595520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.595103 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kmbx4\"" Sep 16 13:43:59.595520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.595353 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 13:43:59.628246 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.628220 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frj4k" Sep 16 13:43:59.673627 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.673591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.733236 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.733203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgvk\" (UniqueName: \"kubernetes.io/projected/6052c427-0ff6-48e3-a6d9-0b14d656e9ea-kube-api-access-rkgvk\") pod \"downloads-76f99584f4-rbxrh\" (UID: \"6052c427-0ff6-48e3-a6d9-0b14d656e9ea\") " pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:43:59.795485 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.795443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.834517 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.834439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgvk\" (UniqueName: \"kubernetes.io/projected/6052c427-0ff6-48e3-a6d9-0b14d656e9ea-kube-api-access-rkgvk\") pod \"downloads-76f99584f4-rbxrh\" (UID: \"6052c427-0ff6-48e3-a6d9-0b14d656e9ea\") " pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:43:59.851070 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.851025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgvk\" (UniqueName: \"kubernetes.io/projected/6052c427-0ff6-48e3-a6d9-0b14d656e9ea-kube-api-access-rkgvk\") pod \"downloads-76f99584f4-rbxrh\" (UID: \"6052c427-0ff6-48e3-a6d9-0b14d656e9ea\") " pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:43:59.875916 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.875858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.903745 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:43:59.903653 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:44:00.034361 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:00.034300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:00.119018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:00.118939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:00.205337 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:00.205254 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:00.316394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:00.316334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.063744 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.063703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.145277 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.145243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.428197 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.428115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.663404 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.663357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.775911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.775857 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.872128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:01.872070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:02.738260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.738223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" event={"ID":"7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3","Type":"ContainerStarted","Data":"aff65c6b5e0ee32a066f62a7726ce920bc4fa81fe2ed331c69fd396c6d55b77d"} Sep 16 13:44:02.739709 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.739686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" event={"ID":"085ba3a6-9ad8-447e-8995-b7241b06e1bd","Type":"ContainerStarted","Data":"78cdb6270edba74a0e5fbf2ebfba67907974c0f838a2d3a05862d3e7e95924a5"} Sep 16 13:44:02.739855 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.739792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:44:02.741481 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.741459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" event={"ID":"924fa136-68e1-4ccb-8fb5-f8d4e677d8b0","Type":"ContainerStarted","Data":"32f33afc9ce6e62a1ac35a144ecff793adf9c0d65ef452c08fc714cafc374901"} Sep 16 13:44:02.741717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.741699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" Sep 16 13:44:02.800604 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.800329 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-w5jsw" podStartSLOduration=1.8296234569999998 podStartE2EDuration="6.800310039s" totalImagesPullingTime="4.970686582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.765003807 +0000 UTC m=+61.805597763" watchObservedRunningTime="2026-09-16 13:44:02.800310039 +0000 UTC m=+61.840903995" Sep 16 13:44:02.800795 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.800495 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7777b69b8d-wv8xk" podStartSLOduration=1.8314524030000001 podStartE2EDuration="6.800488656s" totalImagesPullingTime="4.969036253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.7992742 +0000 UTC m=+61.839868156" watchObservedRunningTime="2026-09-16 13:44:02.800488656 +0000 UTC m=+61.841082611" Sep 16 13:44:02.819422 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.819388 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rbxrh"] Sep 16 13:44:02.823561 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:02.823535 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052c427_0ff6_48e3_a6d9_0b14d656e9ea.slice/crio-a4752a637dfb99ab968d34c7ee343ffb56e2f3d8dde9f68f5379c94bf1167d5c WatchSource:0}: Error finding container a4752a637dfb99ab968d34c7ee343ffb56e2f3d8dde9f68f5379c94bf1167d5c: Status 404 returned error can't find the container with id a4752a637dfb99ab968d34c7ee343ffb56e2f3d8dde9f68f5379c94bf1167d5c Sep 16 13:44:02.833670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.831920 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55fcf47d6d-hgj7f" podStartSLOduration=1.865119043 podStartE2EDuration="6.831897978s" totalImagesPullingTime="4.966778935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.830708175 +0000 UTC m=+61.871302129" watchObservedRunningTime="2026-09-16 13:44:02.831897978 +0000 UTC m=+61.872491934" Sep 16 13:44:02.925676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:02.925630 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:03.043765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.043723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:03.087725 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.087645 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8"] Sep 16 13:44:03.092038 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.092014 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.094396 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.094373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 13:44:03.094535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.094454 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d9d8p\"" Sep 16 13:44:03.094908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.094884 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 13:44:03.095032 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.094931 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 13:44:03.095032 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.094933 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 13:44:03.095282 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.095269 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 13:44:03.102892 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.102843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8"] Sep 16 13:44:03.132144 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.132100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:03.160296 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.160255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.160463 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.160325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3869572-0c46-4889-916a-e691800e3b86-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.160463 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.160358 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.160463 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.160449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhss\" (UniqueName: \"kubernetes.io/projected/c3869572-0c46-4889-916a-e691800e3b86-kube-api-access-rfhss\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.261184 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.261145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhss\" (UniqueName: \"kubernetes.io/projected/c3869572-0c46-4889-916a-e691800e3b86-kube-api-access-rfhss\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.261351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.261207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.261351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.261280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3869572-0c46-4889-916a-e691800e3b86-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.261351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.261314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.261463 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:44:03.261316 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 13:44:03.261463 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:44:03.261418 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls podName:c3869572-0c46-4889-916a-e691800e3b86 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:03.761399226 +0000 UTC m=+62.801993175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-kv7g8" (UID: "c3869572-0c46-4889-916a-e691800e3b86") : secret "prometheus-operator-tls" not found Sep 16 13:44:03.261943 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.261906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3869572-0c46-4889-916a-e691800e3b86-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.263796 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.263777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.274599 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.274570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhss\" (UniqueName: \"kubernetes.io/projected/c3869572-0c46-4889-916a-e691800e3b86-kube-api-access-rfhss\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.497621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.497589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:03.746604 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.746566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rbxrh" event={"ID":"6052c427-0ff6-48e3-a6d9-0b14d656e9ea","Type":"ContainerStarted","Data":"a4752a637dfb99ab968d34c7ee343ffb56e2f3d8dde9f68f5379c94bf1167d5c"} Sep 16 13:44:03.766515 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.766438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:03.769253 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:03.769210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3869572-0c46-4889-916a-e691800e3b86-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kv7g8\" (UID: \"c3869572-0c46-4889-916a-e691800e3b86\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:04.000862 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:04.000809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" Sep 16 13:44:04.289969 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:04.289930 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8"] Sep 16 13:44:04.293378 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:04.293336 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3869572_0c46_4889_916a_e691800e3b86.slice/crio-5f66c815908f35671a0bc2507674418455391d50b624920b5fa75bd25c05291a WatchSource:0}: Error finding container 5f66c815908f35671a0bc2507674418455391d50b624920b5fa75bd25c05291a: Status 404 returned error can't find the container with id 5f66c815908f35671a0bc2507674418455391d50b624920b5fa75bd25c05291a Sep 16 13:44:04.751609 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:04.751574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" event={"ID":"c3869572-0c46-4889-916a-e691800e3b86","Type":"ContainerStarted","Data":"5f66c815908f35671a0bc2507674418455391d50b624920b5fa75bd25c05291a"} Sep 16 13:44:05.757330 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:05.757252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" event={"ID":"c3869572-0c46-4889-916a-e691800e3b86","Type":"ContainerStarted","Data":"a88a37fc76b257b24f470ff097894404a9330a0479f36fadff85e8065c13bd5d"} Sep 16 13:44:06.115089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.115053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.283629 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.283591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.543015 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.542975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.696843 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.696809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.764245 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.764209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" event={"ID":"c3869572-0c46-4889-916a-e691800e3b86","Type":"ContainerStarted","Data":"cdfc0008380c1c45fc6baf6808b994e462024f46498b62cd6c1c9b6ad17753c0"} Sep 16 13:44:06.791229 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:06.791170 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kv7g8" podStartSLOduration=2.456399768 podStartE2EDuration="3.791151775s" totalImagesPullingTime="1.334752007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:06.789606702 +0000 UTC m=+65.830200664" watchObservedRunningTime="2026-09-16 13:44:06.791151775 +0000 UTC m=+65.831745730" Sep 16 13:44:07.057522 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.057483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.153653 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.153616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.298019 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.297974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:44:07.298193 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.298029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:44:07.302107 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.302078 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:44:07.302635 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.302615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:44:07.311480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.311413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0-original-pull-secret\") pod \"global-pull-secret-syncer-8stjr\" (UID: \"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0\") " pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:44:07.311613 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.311489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dffacf5a-422a-42f5-a9c4-65f769ed48ab-metrics-certs\") pod \"network-metrics-daemon-cwjfb\" (UID: \"dffacf5a-422a-42f5-a9c4-65f769ed48ab\") " pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:44:07.399107 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.399065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:44:07.403058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.403030 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:44:07.412341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.412312 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:44:07.423097 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.423064 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9hf\" (UniqueName: \"kubernetes.io/projected/1a148477-2b87-4b48-bde5-b0f21b9b2ac9-kube-api-access-8c9hf\") pod \"network-check-target-6d6sc\" (UID: \"1a148477-2b87-4b48-bde5-b0f21b9b2ac9\") " pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:44:07.576171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.576081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8stjr" Sep 16 13:44:07.584250 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.584220 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\"" Sep 16 13:44:07.588982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.588945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\"" Sep 16 13:44:07.592262 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.592238 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwjfb" Sep 16 13:44:07.596468 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.596447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:44:07.901478 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.901402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8stjr"] Sep 16 13:44:07.905317 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:07.905281 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0af04b0_d57c_4f69_bfdb_ad22dcdfe3e0.slice/crio-2068ebc8562d4432273baa1428d2b65e1bfdb41d9c9bb62788e6bf4e44d5ab56 WatchSource:0}: Error finding container 2068ebc8562d4432273baa1428d2b65e1bfdb41d9c9bb62788e6bf4e44d5ab56: Status 404 returned error can't find the container with id 2068ebc8562d4432273baa1428d2b65e1bfdb41d9c9bb62788e6bf4e44d5ab56 Sep 16 13:44:07.952512 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.952419 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cwjfb"] Sep 16 13:44:07.956805 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:07.956772 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffacf5a_422a_42f5_a9c4_65f769ed48ab.slice/crio-81ebb28c6a5cd5559e43c7d550bfca4acf1ff6ee620dbb873fac2842c8227f80 WatchSource:0}: Error finding container 81ebb28c6a5cd5559e43c7d550bfca4acf1ff6ee620dbb873fac2842c8227f80: Status 404 returned error can't find the container with id 81ebb28c6a5cd5559e43c7d550bfca4acf1ff6ee620dbb873fac2842c8227f80 Sep 16 13:44:07.999667 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:07.999631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6d6sc"] Sep 16 13:44:08.002969 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:08.002942 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a148477_2b87_4b48_bde5_b0f21b9b2ac9.slice/crio-9490c5dd9eea82830dbf494bb55ec2320be55624699931c23d2e6d6636fdff19 WatchSource:0}: Error finding container 9490c5dd9eea82830dbf494bb55ec2320be55624699931c23d2e6d6636fdff19: Status 404 returned error can't find the container with id 9490c5dd9eea82830dbf494bb55ec2320be55624699931c23d2e6d6636fdff19 Sep 16 13:44:08.295345 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.295309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.548680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.548600 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.662736 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.662694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.774501 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.774463 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8stjr" event={"ID":"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0","Type":"ContainerStarted","Data":"2068ebc8562d4432273baa1428d2b65e1bfdb41d9c9bb62788e6bf4e44d5ab56"} Sep 16 13:44:08.775969 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.775896 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6d6sc" event={"ID":"1a148477-2b87-4b48-bde5-b0f21b9b2ac9","Type":"ContainerStarted","Data":"9490c5dd9eea82830dbf494bb55ec2320be55624699931c23d2e6d6636fdff19"} Sep 16 13:44:08.777579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.777523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwjfb" event={"ID":"dffacf5a-422a-42f5-a9c4-65f769ed48ab","Type":"ContainerStarted","Data":"81ebb28c6a5cd5559e43c7d550bfca4acf1ff6ee620dbb873fac2842c8227f80"} Sep 16 13:44:08.794038 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.794000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.847296 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.845769 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg"] Sep 16 13:44:08.848510 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.848482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:08.854601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.853759 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 13:44:08.854601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.854159 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fr2ns\"" Sep 16 13:44:08.854601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.854429 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 13:44:08.873980 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.873945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg"] Sep 16 13:44:08.909592 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.909552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:08.910198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.909614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jk8h\" (UniqueName: \"kubernetes.io/projected/5697cc5a-2e33-44dd-aacf-ebee4d46f996-kube-api-access-5jk8h\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:08.910198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.909679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697cc5a-2e33-44dd-aacf-ebee4d46f996-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:08.910198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.909716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:08.920844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.920804 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tlhzp"] Sep 16 13:44:08.923433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.923402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:08.926334 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.926296 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 13:44:08.926977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.926587 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 13:44:08.926977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.926728 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 13:44:08.926977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.926823 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5m2g2\"" Sep 16 13:44:08.957896 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:08.957840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.010318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-textfile\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010478 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697cc5a-2e33-44dd-aacf-ebee4d46f996-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.010478 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.010478 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-sys\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-wtmp\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.010752 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jk8h\" (UniqueName: \"kubernetes.io/projected/5697cc5a-2e33-44dd-aacf-ebee4d46f996-kube-api-access-5jk8h\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.010806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-root\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010925 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010818 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-tls\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010925 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cbh\" (UniqueName: \"kubernetes.io/projected/f6011656-7048-4f10-8b3d-66a28455f59e-kube-api-access-l2cbh\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.010925 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.010899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-metrics-client-ca\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.011573 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.011548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697cc5a-2e33-44dd-aacf-ebee4d46f996-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.013774 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.013742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.018104 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.018041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5697cc5a-2e33-44dd-aacf-ebee4d46f996-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.037709 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.037620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jk8h\" (UniqueName: \"kubernetes.io/projected/5697cc5a-2e33-44dd-aacf-ebee4d46f996-kube-api-access-5jk8h\") pod \"openshift-state-metrics-67dcf8c979-2pjlg\" (UID: \"5697cc5a-2e33-44dd-aacf-ebee4d46f996\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.104345 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.104263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.112149 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-sys\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-wtmp\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-wtmp\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-root\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-tls\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cbh\" (UniqueName: \"kubernetes.io/projected/f6011656-7048-4f10-8b3d-66a28455f59e-kube-api-access-l2cbh\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-metrics-client-ca\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.112730 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-textfile\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.113431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112897 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-textfile\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.113431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.113431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-sys\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.113431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.112998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6011656-7048-4f10-8b3d-66a28455f59e-root\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.113431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.113385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6011656-7048-4f10-8b3d-66a28455f59e-metrics-client-ca\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.116573 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.116543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.118837 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.118786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6011656-7048-4f10-8b3d-66a28455f59e-node-exporter-tls\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.147576 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.147543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cbh\" (UniqueName: \"kubernetes.io/projected/f6011656-7048-4f10-8b3d-66a28455f59e-kube-api-access-l2cbh\") pod \"node-exporter-tlhzp\" (UID: \"f6011656-7048-4f10-8b3d-66a28455f59e\") " pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.164351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.164315 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" Sep 16 13:44:09.237900 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.237854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlhzp" Sep 16 13:44:09.258717 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:09.258678 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6011656_7048_4f10_8b3d_66a28455f59e.slice/crio-4db4a98e88d1c0dfb96a8f744058d271e032908243676e5f17189292e96704b0 WatchSource:0}: Error finding container 4db4a98e88d1c0dfb96a8f744058d271e032908243676e5f17189292e96704b0: Status 404 returned error can't find the container with id 4db4a98e88d1c0dfb96a8f744058d271e032908243676e5f17189292e96704b0 Sep 16 13:44:09.302183 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.302094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.443171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.443100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.490585 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.490491 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg"] Sep 16 13:44:09.495039 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:09.495001 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5697cc5a_2e33_44dd_aacf_ebee4d46f996.slice/crio-c8d3543bb160b9021e4d42cc044b56052ee104fd68f8ddcea12571e4c2d1046a WatchSource:0}: Error finding container c8d3543bb160b9021e4d42cc044b56052ee104fd68f8ddcea12571e4c2d1046a: Status 404 returned error can't find the container with id c8d3543bb160b9021e4d42cc044b56052ee104fd68f8ddcea12571e4c2d1046a Sep 16 13:44:09.721010 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.720847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.785393 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.785174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlhzp" event={"ID":"f6011656-7048-4f10-8b3d-66a28455f59e","Type":"ContainerStarted","Data":"4db4a98e88d1c0dfb96a8f744058d271e032908243676e5f17189292e96704b0"} Sep 16 13:44:09.788111 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.788074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" event={"ID":"5697cc5a-2e33-44dd-aacf-ebee4d46f996","Type":"ContainerStarted","Data":"6853464929cf43f912c1ab51afe0639d8c38cee221c5de8eb9d144607814f893"} Sep 16 13:44:09.788309 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.788136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" event={"ID":"5697cc5a-2e33-44dd-aacf-ebee4d46f996","Type":"ContainerStarted","Data":"44364d675a057548cd55d16c63c95fe0d0de704fa25082257d7d7c5141c209be"} Sep 16 13:44:09.788309 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.788164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" event={"ID":"5697cc5a-2e33-44dd-aacf-ebee4d46f996","Type":"ContainerStarted","Data":"c8d3543bb160b9021e4d42cc044b56052ee104fd68f8ddcea12571e4c2d1046a"} Sep 16 13:44:09.792470 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.792366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwjfb" event={"ID":"dffacf5a-422a-42f5-a9c4-65f769ed48ab","Type":"ContainerStarted","Data":"6363c278d00ec86e3801678fc74f7ecb624aca19b15f58ebb01387ac23f282c8"} Sep 16 13:44:09.792470 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.792432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwjfb" event={"ID":"dffacf5a-422a-42f5-a9c4-65f769ed48ab","Type":"ContainerStarted","Data":"96dafdd7b9ee58f6534c0c858f053350beba2506f27cb736a90a6a364e9c0a25"} Sep 16 13:44:09.930299 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:09.930211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.047054 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.046837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.121667 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.120023 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cwjfb" podStartSLOduration=68.001708781 podStartE2EDuration="1m9.119996282s" totalImagesPullingTime="1.118287501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:09.843202307 +0000 UTC m=+68.883796265" watchObservedRunningTime="2026-09-16 13:44:10.119996282 +0000 UTC m=+69.160590241" Sep 16 13:44:10.124645 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.122487 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:44:10.128860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.126839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.129939 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.129861 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.130844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.131124 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.131377 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.131552 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.131797 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b9qcg\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.131802 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.132006 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 13:44:10.132063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.132039 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 13:44:10.143322 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.142954 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 13:44:10.164424 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.164306 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:44:10.222051 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.221935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223097 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbmr\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.223701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.223527 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.324904 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.324957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbmr\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326110 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.325330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.326931 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.326194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.328547 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.328177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.329680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.329649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.332147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.331993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.332147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.332055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.332147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.332104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.332969 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.332883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.332969 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.332923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.333555 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.333512 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.333824 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.333797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.336529 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.336477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.337002 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.336981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.343679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.343576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbmr\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr\") pod \"alertmanager-main-0\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.356485 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.356359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.444856 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.444808 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:44:10.552033 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.551929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.784046 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.783969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:44:10.787824 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:10.787788 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c0c476_715f_4e5b_b462_581d32225e60.slice/crio-953f097dd4a9ee9bbe024ff6ce52f640ca5d04b1ee7c2ced94f04f447943b35d WatchSource:0}: Error finding container 953f097dd4a9ee9bbe024ff6ce52f640ca5d04b1ee7c2ced94f04f447943b35d: Status 404 returned error can't find the container with id 953f097dd4a9ee9bbe024ff6ce52f640ca5d04b1ee7c2ced94f04f447943b35d Sep 16 13:44:10.800760 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.800724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"953f097dd4a9ee9bbe024ff6ce52f640ca5d04b1ee7c2ced94f04f447943b35d"} Sep 16 13:44:10.805084 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.804909 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6011656-7048-4f10-8b3d-66a28455f59e" containerID="a57236d418663318171d72e3bbda3d5850ff6589830abbbdf328c010ee0af06b" exitCode=0 Sep 16 13:44:10.805084 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.805012 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlhzp" event={"ID":"f6011656-7048-4f10-8b3d-66a28455f59e","Type":"ContainerDied","Data":"a57236d418663318171d72e3bbda3d5850ff6589830abbbdf328c010ee0af06b"} Sep 16 13:44:10.898968 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:10.898929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.009097 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:11.009060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.812888 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:11.812829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlhzp" event={"ID":"f6011656-7048-4f10-8b3d-66a28455f59e","Type":"ContainerStarted","Data":"730752f3e53adaa55b6733d60d9dfd85643851b9982b2ccb2a9141285012c183"} Sep 16 13:44:12.279820 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:12.279773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.404602 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:12.404560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.588848 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:12.588754 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.052789 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.052750 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69c87f9bd9-nrl67"] Sep 16 13:44:13.057020 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.056994 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.059601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.059569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 13:44:13.059747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.059714 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d5urs5slpv0b7\"" Sep 16 13:44:13.059966 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.059934 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 13:44:13.060061 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.060004 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 13:44:13.060112 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.059950 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 13:44:13.060364 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.060204 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 13:44:13.060364 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.060344 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-z9946\"" Sep 16 13:44:13.068765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.068740 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c87f9bd9-nrl67"] Sep 16 13:44:13.156106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8ead6e1-f509-4287-8261-d6da0a52c463-metrics-client-ca\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncgf\" (UniqueName: \"kubernetes.io/projected/a8ead6e1-f509-4287-8261-d6da0a52c463-kube-api-access-qncgf\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-grpc-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.156579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.156426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257059 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257233 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257233 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257233 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8ead6e1-f509-4287-8261-d6da0a52c463-metrics-client-ca\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257233 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncgf\" (UniqueName: \"kubernetes.io/projected/a8ead6e1-f509-4287-8261-d6da0a52c463-kube-api-access-qncgf\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257233 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257482 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-grpc-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.257748 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.257651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.258266 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.258239 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8ead6e1-f509-4287-8261-d6da0a52c463-metrics-client-ca\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.260583 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.260523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.262841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.261233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.262841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.262281 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.262841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.262401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.262841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.262405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.262841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.262801 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8ead6e1-f509-4287-8261-d6da0a52c463-secret-grpc-tls\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.276067 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.275028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncgf\" (UniqueName: \"kubernetes.io/projected/a8ead6e1-f509-4287-8261-d6da0a52c463-kube-api-access-qncgf\") pod \"thanos-querier-69c87f9bd9-nrl67\" (UID: \"a8ead6e1-f509-4287-8261-d6da0a52c463\") " pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.329017 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.328921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.371284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.371240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:13.432715 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.432606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.515182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:13.515147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:14.505523 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:14.505473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:14.602663 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:14.602629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:15.135125 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:15.135082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:15.235098 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:15.235065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:15.605075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:15.605032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:19.732198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:19.732169 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b5f48df84-4rfvr" Sep 16 13:44:19.928541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:19.928507 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:20.031213 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:20.031135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.244554 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.244514 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.349025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.348987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.666755 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.666690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c87f9bd9-nrl67"] Sep 16 13:44:21.858278 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.858192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlhzp" event={"ID":"f6011656-7048-4f10-8b3d-66a28455f59e","Type":"ContainerStarted","Data":"fd7fa65e7ead539d008fd7933df932ba158d6666ce3baa1bac1cfa87339f76f7"} Sep 16 13:44:21.860772 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.860728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" event={"ID":"5697cc5a-2e33-44dd-aacf-ebee4d46f996","Type":"ContainerStarted","Data":"2c424846ddc3db17f6358e44700576ba3e44729516a580d5855c6d5c0050e521"} Sep 16 13:44:21.862490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.862452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6d6sc" event={"ID":"1a148477-2b87-4b48-bde5-b0f21b9b2ac9","Type":"ContainerStarted","Data":"940fdf01a5036371b361c2878701cb7b64792152321419ecbd2e9cb1f1e2b203"} Sep 16 13:44:21.862655 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.862520 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:44:21.869581 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.869529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.881050 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.880969 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tlhzp" podStartSLOduration=12.623098151 podStartE2EDuration="13.880943884s" totalImagesPullingTime="1.169304961s" totalInitContainerRuntime="88.540772ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:21.880475186 +0000 UTC m=+80.921069165" watchObservedRunningTime="2026-09-16 13:44:21.880943884 +0000 UTC m=+80.921537842" Sep 16 13:44:21.910611 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.910556 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2pjlg" podStartSLOduration=2.211975215 podStartE2EDuration="13.910538097s" totalImagesPullingTime="11.698562882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:21.909354084 +0000 UTC m=+80.949948038" watchObservedRunningTime="2026-09-16 13:44:21.910538097 +0000 UTC m=+80.951132045" Sep 16 13:44:21.929418 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.929364 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6d6sc" podStartSLOduration=67.543696942 podStartE2EDuration="1m20.929347558s" totalImagesPullingTime="13.385650616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:21.928939401 +0000 UTC m=+80.969533346" watchObservedRunningTime="2026-09-16 13:44:21.929347558 +0000 UTC m=+80.969941513" Sep 16 13:44:21.949162 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:44:21.949125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ead6e1_f509_4287_8261_d6da0a52c463.slice/crio-33123e667d1b74eba68d5d0de4fe7250500ff2fa4861ab4170f0989298f3a9aa WatchSource:0}: Error finding container 33123e667d1b74eba68d5d0de4fe7250500ff2fa4861ab4170f0989298f3a9aa: Status 404 returned error can't find the container with id 33123e667d1b74eba68d5d0de4fe7250500ff2fa4861ab4170f0989298f3a9aa Sep 16 13:44:21.988673 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:21.988643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.086161 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.086138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.454214 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.454158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.524524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.524486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.871696 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.871321 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07" exitCode=0 Sep 16 13:44:22.871696 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.871483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07"} Sep 16 13:44:22.874733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.874392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rbxrh" event={"ID":"6052c427-0ff6-48e3-a6d9-0b14d656e9ea","Type":"ContainerStarted","Data":"f0bf896fa0b558706fcef01a8e83e4d543630fd17ddb102173250550e695580e"} Sep 16 13:44:22.875202 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.875136 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:44:22.877465 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.877432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"33123e667d1b74eba68d5d0de4fe7250500ff2fa4861ab4170f0989298f3a9aa"} Sep 16 13:44:22.888436 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.888406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-rbxrh" Sep 16 13:44:22.890204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.890165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8stjr" event={"ID":"c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0","Type":"ContainerStarted","Data":"226c548b6f3308215d7d4e5432816109cff7a18f8ff4a061facdb8b1aafe59bc"} Sep 16 13:44:22.931738 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.931410 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8stjr" podStartSLOduration=67.871955334 podStartE2EDuration="1m21.93138901s" totalImagesPullingTime="14.059433676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:22.931271895 +0000 UTC m=+81.971865861" watchObservedRunningTime="2026-09-16 13:44:22.93138901 +0000 UTC m=+81.971982963" Sep 16 13:44:22.953472 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:22.953391 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-rbxrh" podStartSLOduration=4.761462169 podStartE2EDuration="23.953370524s" totalImagesPullingTime="19.191908355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:22.952585012 +0000 UTC m=+81.993178968" watchObservedRunningTime="2026-09-16 13:44:22.953370524 +0000 UTC m=+81.993964481" Sep 16 13:44:24.073523 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:24.073361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.188039 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:24.187988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.270776 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:24.270714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:25.915561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.915232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6"} Sep 16 13:44:25.915561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.915365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840"} Sep 16 13:44:25.915561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.915413 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c"} Sep 16 13:44:25.915561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.915455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9"} Sep 16 13:44:25.915561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.915491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6"} Sep 16 13:44:25.918067 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.918038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"d7c7f22e42ff229c93a77a2b1a2324a3ded77b58e500efcd557336459bcf830b"} Sep 16 13:44:25.918202 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.918113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"2e95c88471272305b877110ffe3b51d73ccfee93749a53acd086a1de2b2b98e5"} Sep 16 13:44:25.918202 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:25.918154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"181a75e4e875aa56de0bb059487cf54518cde01f583ce6b35aaf5714e7d0d010"} Sep 16 13:44:27.934157 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.934117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerStarted","Data":"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1"} Sep 16 13:44:27.937682 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.937654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"205e86d93a6a7beae7d310f7341e0fb1d2d2f07b866a0964e32036bff9d5d752"} Sep 16 13:44:27.937813 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.937725 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"426446182b1a256c44f0b2097a2ed063e2211daa52a915571eb3d22f1f701e09"} Sep 16 13:44:27.937813 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.937768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" event={"ID":"a8ead6e1-f509-4287-8261-d6da0a52c463","Type":"ContainerStarted","Data":"2f9775b5a68a7203676da8aa9eb258d10c418e262ceb22d6df05e64f95e8c489"} Sep 16 13:44:27.937935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.937913 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:27.969832 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.969699 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.352450885 podStartE2EDuration="17.96967898s" totalImagesPullingTime="14.60092388s" totalInitContainerRuntime="16.304215ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:27.969027641 +0000 UTC m=+87.009621597" watchObservedRunningTime="2026-09-16 13:44:27.96967898 +0000 UTC m=+87.010272937" Sep 16 13:44:27.999291 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:27.999044 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" podStartSLOduration=10.500582519 podStartE2EDuration="14.999019931s" totalImagesPullingTime="4.498437412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:27.998275119 +0000 UTC m=+87.038869087" watchObservedRunningTime="2026-09-16 13:44:27.999019931 +0000 UTC m=+87.039613887" Sep 16 13:44:29.500413 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:29.500346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:29.580530 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:29.580488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.320935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:32.320893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.382703 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:32.382667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.458007 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:32.457973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.560126 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:32.560085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:33.948599 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:33.948572 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69c87f9bd9-nrl67" Sep 16 13:44:44.493893 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:44.493839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:44.589498 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:44.589443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.892568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:44:52.892536 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6d6sc" Sep 16 13:45:02.130041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:02.129998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.505859 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:04.505821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:08.276520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:08.276483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:08.352670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:08.352632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:25.406853 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:25.406819 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:25.479806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:25.479775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:29.772300 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772256 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:29.772692 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772602 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="alertmanager" containerID="cri-o://01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" gracePeriod=120 Sep 16 13:45:29.772692 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772658 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-metric" containerID="cri-o://7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" gracePeriod=120 Sep 16 13:45:29.772804 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772688 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-web" containerID="cri-o://706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" gracePeriod=120 Sep 16 13:45:29.772804 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772737 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy" containerID="cri-o://a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" gracePeriod=120 Sep 16 13:45:29.772804 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772738 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="prom-label-proxy" containerID="cri-o://df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" gracePeriod=120 Sep 16 13:45:29.772977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:29.772731 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="config-reloader" containerID="cri-o://b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" gracePeriod=120 Sep 16 13:45:30.193826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193797 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" exitCode=0 Sep 16 13:45:30.193826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193823 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" exitCode=0 Sep 16 13:45:30.193826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193829 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" exitCode=0 Sep 16 13:45:30.194080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193835 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" exitCode=0 Sep 16 13:45:30.194080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1"} Sep 16 13:45:30.194080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840"} Sep 16 13:45:30.194080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.193988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9"} Sep 16 13:45:30.194080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:30.194013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6"} Sep 16 13:45:31.014508 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.014479 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.117253 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117168 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117253 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117208 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117253 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117243 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117268 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117336 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117361 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117389 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117416 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117447 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117497 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117991 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117541 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsbmr\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr\") pod \"31c0c476-715f-4e5b-b462-581d32225e60\" (UID: \"31c0c476-715f-4e5b-b462-581d32225e60\") " Sep 16 13:45:31.117991 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117702 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:31.117991 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.117811 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.119141 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.118525 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:31.119141 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.118858 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:31.121464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.121409 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.122002 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.121973 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.122908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.122838 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.122908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.122852 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr" (OuterVolumeSpecName: "kube-api-access-bsbmr") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "kube-api-access-bsbmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:31.123184 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.123094 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.123184 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.123135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out" (OuterVolumeSpecName: "config-out") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:31.123310 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.123232 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume" (OuterVolumeSpecName: "config-volume") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.123310 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.123270 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:31.125483 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.125458 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.132705 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.132675 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config" (OuterVolumeSpecName: "web-config") pod "31c0c476-715f-4e5b-b462-581d32225e60" (UID: "31c0c476-715f-4e5b-b462-581d32225e60"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:31.200739 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200707 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" exitCode=0 Sep 16 13:45:31.200739 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200733 2559 generic.go:356] "Generic (PLEG): container finished" podID="31c0c476-715f-4e5b-b462-581d32225e60" containerID="706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" exitCode=0 Sep 16 13:45:31.200959 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6"} Sep 16 13:45:31.200959 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.200959 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c"} Sep 16 13:45:31.200959 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31c0c476-715f-4e5b-b462-581d32225e60","Type":"ContainerDied","Data":"953f097dd4a9ee9bbe024ff6ce52f640ca5d04b1ee7c2ced94f04f447943b35d"} Sep 16 13:45:31.200959 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.200892 2559 scope.go:117] "RemoveContainer" containerID="df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" Sep 16 13:45:31.209222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.209200 2559 scope.go:117] "RemoveContainer" containerID="7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" Sep 16 13:45:31.216948 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.216922 2559 scope.go:117] "RemoveContainer" containerID="a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" Sep 16 13:45:31.218352 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218328 2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218356 2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-cluster-tls-config\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218369 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-web-config\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218388 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-alertmanager-main-db\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218404 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218418 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31c0c476-715f-4e5b-b462-581d32225e60-config-out\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218429 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218439 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c0c476-715f-4e5b-b462-581d32225e60-metrics-client-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218448 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-tls-assets\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218456 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-main-tls\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218468 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbmr\" (UniqueName: \"kubernetes.io/projected/31c0c476-715f-4e5b-b462-581d32225e60-kube-api-access-bsbmr\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.218622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.218480 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31c0c476-715f-4e5b-b462-581d32225e60-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:45:31.224220 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.224200 2559 scope.go:117] "RemoveContainer" containerID="706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" Sep 16 13:45:31.227571 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.227535 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:31.233798 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.233775 2559 scope.go:117] "RemoveContainer" containerID="b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" Sep 16 13:45:31.237255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.237227 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:31.242406 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.242385 2559 scope.go:117] "RemoveContainer" containerID="01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" Sep 16 13:45:31.250332 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.250299 2559 scope.go:117] "RemoveContainer" containerID="105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07" Sep 16 13:45:31.262016 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.261981 2559 scope.go:117] "RemoveContainer" containerID="df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" Sep 16 13:45:31.262965 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.262929 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1\": container with ID starting with df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1 not found: ID does not exist" containerID="df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" Sep 16 13:45:31.263067 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.262976 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1"} err="failed to get container status \"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1\": rpc error: code = NotFound desc = could not find container \"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1\": container with ID starting with df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1 not found: ID does not exist" Sep 16 13:45:31.263067 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263021 2559 scope.go:117] "RemoveContainer" containerID="7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" Sep 16 13:45:31.263385 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.263359 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6\": container with ID starting with 7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6 not found: ID does not exist" containerID="7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" Sep 16 13:45:31.263466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263391 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6"} err="failed to get container status \"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6\": rpc error: code = NotFound desc = could not find container \"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6\": container with ID starting with 7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6 not found: ID does not exist" Sep 16 13:45:31.263466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263408 2559 scope.go:117] "RemoveContainer" containerID="a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" Sep 16 13:45:31.263688 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.263672 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840\": container with ID starting with a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840 not found: ID does not exist" containerID="a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" Sep 16 13:45:31.263740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263699 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840"} err="failed to get container status \"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840\": rpc error: code = NotFound desc = could not find container \"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840\": container with ID starting with a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840 not found: ID does not exist" Sep 16 13:45:31.263740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263713 2559 scope.go:117] "RemoveContainer" containerID="706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" Sep 16 13:45:31.263995 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.263973 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c\": container with ID starting with 706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c not found: ID does not exist" containerID="706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" Sep 16 13:45:31.264111 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.263996 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c"} err="failed to get container status \"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c\": rpc error: code = NotFound desc = could not find container \"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c\": container with ID starting with 706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c not found: ID does not exist" Sep 16 13:45:31.264111 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264011 2559 scope.go:117] "RemoveContainer" containerID="b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" Sep 16 13:45:31.264293 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.264274 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9\": container with ID starting with b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9 not found: ID does not exist" containerID="b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" Sep 16 13:45:31.264349 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264298 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9"} err="failed to get container status \"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9\": rpc error: code = NotFound desc = could not find container \"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9\": container with ID starting with b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9 not found: ID does not exist" Sep 16 13:45:31.264349 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264313 2559 scope.go:117] "RemoveContainer" containerID="01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" Sep 16 13:45:31.264611 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.264593 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6\": container with ID starting with 01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6 not found: ID does not exist" containerID="01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" Sep 16 13:45:31.264674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264615 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6"} err="failed to get container status \"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6\": rpc error: code = NotFound desc = could not find container \"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6\": container with ID starting with 01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6 not found: ID does not exist" Sep 16 13:45:31.264674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264631 2559 scope.go:117] "RemoveContainer" containerID="105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07" Sep 16 13:45:31.264853 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:45:31.264837 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07\": container with ID starting with 105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07 not found: ID does not exist" containerID="105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07" Sep 16 13:45:31.264939 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264856 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07"} err="failed to get container status \"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07\": rpc error: code = NotFound desc = could not find container \"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07\": container with ID starting with 105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07 not found: ID does not exist" Sep 16 13:45:31.264939 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.264892 2559 scope.go:117] "RemoveContainer" containerID="df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1" Sep 16 13:45:31.265188 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265168 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1"} err="failed to get container status \"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1\": rpc error: code = NotFound desc = could not find container \"df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1\": container with ID starting with df398135f99ba7e4a5e54ec66db0b57235de14364446ebac8273c43fee781bd1 not found: ID does not exist" Sep 16 13:45:31.265249 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265188 2559 scope.go:117] "RemoveContainer" containerID="7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6" Sep 16 13:45:31.265405 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265383 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6"} err="failed to get container status \"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6\": rpc error: code = NotFound desc = could not find container \"7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6\": container with ID starting with 7329938a385842260a86efdf78ae86297fea434a5e9eb6ba5af812409c8500a6 not found: ID does not exist" Sep 16 13:45:31.265466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265406 2559 scope.go:117] "RemoveContainer" containerID="a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840" Sep 16 13:45:31.265667 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265645 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840"} err="failed to get container status \"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840\": rpc error: code = NotFound desc = could not find container \"a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840\": container with ID starting with a24bb3cd2d04d927660a079af0b3af5748829922844f9b493cd5273cea137840 not found: ID does not exist" Sep 16 13:45:31.265731 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265666 2559 scope.go:117] "RemoveContainer" containerID="706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c" Sep 16 13:45:31.265947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265924 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c"} err="failed to get container status \"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c\": rpc error: code = NotFound desc = could not find container \"706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c\": container with ID starting with 706d8aa5f63a6e156a5fe21b6bdb93296d432b32fd12890518a42bb1fa70a01c not found: ID does not exist" Sep 16 13:45:31.266018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.265947 2559 scope.go:117] "RemoveContainer" containerID="b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9" Sep 16 13:45:31.266199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.266176 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9"} err="failed to get container status \"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9\": rpc error: code = NotFound desc = could not find container \"b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9\": container with ID starting with b6307b083e33b22fdc96f4ff60854e3a040fc6964067beb69ebfb46c95241de9 not found: ID does not exist" Sep 16 13:45:31.266254 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.266200 2559 scope.go:117] "RemoveContainer" containerID="01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6" Sep 16 13:45:31.266399 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.266384 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6"} err="failed to get container status \"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6\": rpc error: code = NotFound desc = could not find container \"01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6\": container with ID starting with 01d276fc2262ae3190baafc9ca94af3fc5391ede1ced82228d696a8acb51a4f6 not found: ID does not exist" Sep 16 13:45:31.266463 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.266399 2559 scope.go:117] "RemoveContainer" containerID="105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07" Sep 16 13:45:31.266635 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.266617 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07"} err="failed to get container status \"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07\": rpc error: code = NotFound desc = could not find container \"105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07\": container with ID starting with 105834ac27ebdfd6d8697c2675b51e767750307cdcbde2460a69e5667b2abe07 not found: ID does not exist" Sep 16 13:45:31.270309 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270279 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:31.270711 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270695 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-web" Sep 16 13:45:31.270767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270714 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-web" Sep 16 13:45:31.270767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270727 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="prom-label-proxy" Sep 16 13:45:31.270767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270735 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="prom-label-proxy" Sep 16 13:45:31.270767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270756 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-metric" Sep 16 13:45:31.270767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270765 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-metric" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270780 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="config-reloader" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270788 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="config-reloader" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270799 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="init-config-reloader" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270811 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="init-config-reloader" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270821 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="alertmanager" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270830 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="alertmanager" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270839 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270847 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270924 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="config-reloader" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270935 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-web" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270944 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="alertmanager" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270957 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270967 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="kube-rbac-proxy-metric" Sep 16 13:45:31.270982 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.270975 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c0c476-715f-4e5b-b462-581d32225e60" containerName="prom-label-proxy" Sep 16 13:45:31.275186 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.275161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.277499 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.277473 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 13:45:31.277661 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.277604 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 13:45:31.277661 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.277630 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 13:45:31.277919 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.277900 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 13:45:31.277919 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.277915 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 13:45:31.278093 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.278015 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b9qcg\"" Sep 16 13:45:31.278146 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.278133 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 13:45:31.278199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.278175 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 13:45:31.278362 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.278328 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 13:45:31.283644 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.283621 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 13:45:31.294852 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.292782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:31.420525 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420525 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420525 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-web-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg46k\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-kube-api-access-tg46k\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.420740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-out\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.421018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.421018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.420796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.470200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.470165 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c0c476-715f-4e5b-b462-581d32225e60" path="/var/lib/kubelet/pods/31c0c476-715f-4e5b-b462-581d32225e60/volumes" Sep 16 13:45:31.521742 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.521742 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-web-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.521967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg46k\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-kube-api-access-tg46k\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-out\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522384 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.522828 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.522798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.524844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.524808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-out\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525151 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525151 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525329 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525438 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-web-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525656 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.525785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.525761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74559091-a7b0-46e3-a115-0f5fb03f78e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.526975 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.526958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74559091-a7b0-46e3-a115-0f5fb03f78e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.536446 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.536420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg46k\" (UniqueName: \"kubernetes.io/projected/74559091-a7b0-46e3-a115-0f5fb03f78e0-kube-api-access-tg46k\") pod \"alertmanager-main-0\" (UID: \"74559091-a7b0-46e3-a115-0f5fb03f78e0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.586048 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.586017 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:45:31.872103 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:31.872035 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:45:31.874657 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:45:31.874625 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74559091_a7b0_46e3_a115_0f5fb03f78e0.slice/crio-8f50c6185dd6c7fd02da245b4d37e0742ccc3934bda92b4bc7c8afff7d09f5f8 WatchSource:0}: Error finding container 8f50c6185dd6c7fd02da245b4d37e0742ccc3934bda92b4bc7c8afff7d09f5f8: Status 404 returned error can't find the container with id 8f50c6185dd6c7fd02da245b4d37e0742ccc3934bda92b4bc7c8afff7d09f5f8 Sep 16 13:45:32.206447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:32.206412 2559 generic.go:356] "Generic (PLEG): container finished" podID="74559091-a7b0-46e3-a115-0f5fb03f78e0" containerID="a0a68e7989f1fe4ca51e8a9a18d671621be358fe966603c1b3f586416d2db197" exitCode=0 Sep 16 13:45:32.207035 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:32.206502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerDied","Data":"a0a68e7989f1fe4ca51e8a9a18d671621be358fe966603c1b3f586416d2db197"} Sep 16 13:45:32.207035 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:32.206594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"8f50c6185dd6c7fd02da245b4d37e0742ccc3934bda92b4bc7c8afff7d09f5f8"} Sep 16 13:45:33.216068 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216029 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"3ebd62af82170f425ab48f9fe12f508292d47a2d2c5354cc8718da6aab3db51d"} Sep 16 13:45:33.216509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"a084092b6a2c10023754a8bd7a500b543962766894999193a326472b4474d503"} Sep 16 13:45:33.216509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216162 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"2d9a1e3e42d3a2b4837856624ca2c4545f021916d1ee2f38518fd680bd771444"} Sep 16 13:45:33.216509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"73035930f6378f26c2a0787f901118525777912215bed778751c9909773c5981"} Sep 16 13:45:33.216509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"649779d40cf5fba70a4247e00bbfae5be21edf79963321f4e7a1e4e9fc714f1f"} Sep 16 13:45:33.216509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.216265 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74559091-a7b0-46e3-a115-0f5fb03f78e0","Type":"ContainerStarted","Data":"bbc1f315b048499a83f74660d0d0eae5db49f60815d31b367beca8ebc4502668"} Sep 16 13:45:33.252083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:33.252017 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.236024669 podStartE2EDuration="2.251994712s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.970043ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:33.247026587 +0000 UTC m=+152.287620542" watchObservedRunningTime="2026-09-16 13:45:33.251994712 +0000 UTC m=+152.292588668" Sep 16 13:45:37.369755 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:37.369721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:37.462343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:37.462310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:44.482631 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:44.482589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:44.583995 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:44.583954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:44.647446 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:44.647408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:51.518598 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:51.518565 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:52.156447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:52.156403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:52.233300 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:52.233270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:52.301162 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:52.301127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:52.889699 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:52.889669 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:52.989447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:45:52.989408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:02.642802 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:02.642763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:02.743516 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:02.743483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:02.807313 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:02.807280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:18.273041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:18.273000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:18.343653 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:18.343617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:22.063655 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:22.063617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:22.163467 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:22.163420 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:38.423426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:38.423391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:38.481143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:38.481110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:42.785637 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:42.785605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:52.000853 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:52.000810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:54.020343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:54.020309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:54.993372 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:54.993338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:57.643437 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:57.643406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:57.745906 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:57.745852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:58.997315 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:58.997277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:59.171702 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:59.171665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:59.295145 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:46:59.295060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.449032 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:01.448996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.616745 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:01.616713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:04.488335 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:04.488299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:04.583538 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:04.583504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:08.686062 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:08.686027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.553291 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.553258 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b7smd"] Sep 16 13:47:09.555450 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.555432 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.557730 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.557702 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 13:47:09.557855 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.557739 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d79c2\"" Sep 16 13:47:09.558442 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.558427 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 13:47:09.566100 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.566077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b7smd"] Sep 16 13:47:09.699670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.699630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qspc\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-kube-api-access-7qspc\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.700104 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.699700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-bound-sa-token\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.800816 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.800774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-bound-sa-token\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.801021 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.800862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qspc\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-kube-api-access-7qspc\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.809497 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.809426 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-bound-sa-token\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.809627 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.809563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qspc\" (UniqueName: \"kubernetes.io/projected/160d18e6-dac3-4fe6-bd64-c323c50de54b-kube-api-access-7qspc\") pod \"cert-manager-55b4d954d5-b7smd\" (UID: \"160d18e6-dac3-4fe6-bd64-c323c50de54b\") " pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:09.863680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:09.863630 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-b7smd" Sep 16 13:47:10.138680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:10.138634 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b7smd"] Sep 16 13:47:10.141367 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:47:10.141337 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod160d18e6_dac3_4fe6_bd64_c323c50de54b.slice/crio-6bf77fd0f23d816351dcf415f05c934d96634c56c380531862b1807c6b84e25d WatchSource:0}: Error finding container 6bf77fd0f23d816351dcf415f05c934d96634c56c380531862b1807c6b84e25d: Status 404 returned error can't find the container with id 6bf77fd0f23d816351dcf415f05c934d96634c56c380531862b1807c6b84e25d Sep 16 13:47:10.590218 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:10.590185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-b7smd" event={"ID":"160d18e6-dac3-4fe6-bd64-c323c50de54b","Type":"ContainerStarted","Data":"6bf77fd0f23d816351dcf415f05c934d96634c56c380531862b1807c6b84e25d"} Sep 16 13:47:13.604818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:13.604722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-b7smd" event={"ID":"160d18e6-dac3-4fe6-bd64-c323c50de54b","Type":"ContainerStarted","Data":"40376684aa13997500b7452fedc812f06711a513895437362def556540d5e14c"} Sep 16 13:47:13.628633 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:13.628575 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-b7smd" podStartSLOduration=1.547090822 podStartE2EDuration="4.628558971s" totalImagesPullingTime="3.081468149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:13.626985363 +0000 UTC m=+252.667579317" watchObservedRunningTime="2026-09-16 13:47:13.628558971 +0000 UTC m=+252.669152926" Sep 16 13:47:14.841462 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:14.841417 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:14.937359 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:14.937307 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:21.619163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:21.619129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:21.717844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:21.717812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:23.503036 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:23.503005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:32.015663 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:32.015616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:32.075764 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:32.075728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:35.960008 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:35.959974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:36.067098 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.067066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:36.604019 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.603981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:36.928579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.928496 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx"] Sep 16 13:47:36.935106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.935082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:36.938194 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.938167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 13:47:36.938194 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.938185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n7f75\"" Sep 16 13:47:36.938495 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.938477 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 13:47:36.938535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.938480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 13:47:36.938821 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.938808 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 13:47:36.946932 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:36.946908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx"] Sep 16 13:47:37.007464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.007419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.007464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.007462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.007923 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.007487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.007923 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.007510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntbr\" (UniqueName: \"kubernetes.io/projected/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-kube-api-access-cntbr\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.007923 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.007541 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108174 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108174 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntbr\" (UniqueName: \"kubernetes.io/projected/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-kube-api-access-cntbr\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.108690 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.108664 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.110669 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.110632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.110816 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.110767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.110940 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.110923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.119793 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.119765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntbr\" (UniqueName: \"kubernetes.io/projected/4a92a250-a4cc-4c2f-98ca-c5aa03acc32b-kube-api-access-cntbr\") pod \"opendatahub-operator-controller-manager-f5fcfb689-7sfbx\" (UID: \"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.246637 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.246597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:37.396701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.396662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:37.501777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.501666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:37.526753 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.526724 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx"] Sep 16 13:47:37.529164 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:47:37.529131 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a92a250_a4cc_4c2f_98ca_c5aa03acc32b.slice/crio-1450e905dab73d90cf2ff0ab10a03fb892e12ddcbb8b09d3a5ad4125fcd8f010 WatchSource:0}: Error finding container 1450e905dab73d90cf2ff0ab10a03fb892e12ddcbb8b09d3a5ad4125fcd8f010: Status 404 returned error can't find the container with id 1450e905dab73d90cf2ff0ab10a03fb892e12ddcbb8b09d3a5ad4125fcd8f010 Sep 16 13:47:37.694356 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:37.694310 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" event={"ID":"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b","Type":"ContainerStarted","Data":"1450e905dab73d90cf2ff0ab10a03fb892e12ddcbb8b09d3a5ad4125fcd8f010"} Sep 16 13:47:39.773643 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:39.773605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:39.886562 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:39.886496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:40.709445 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:40.709353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" event={"ID":"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b","Type":"ContainerStarted","Data":"4d6ee4231caf0ff6195e9a7a283a64dbf8701386be08bebd449402ec8edd2532"} Sep 16 13:47:41.715230 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:41.715198 2559 generic.go:356] "Generic (PLEG): container finished" podID="4a92a250-a4cc-4c2f-98ca-c5aa03acc32b" containerID="4d6ee4231caf0ff6195e9a7a283a64dbf8701386be08bebd449402ec8edd2532" exitCode=0 Sep 16 13:47:41.715617 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:41.715241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" event={"ID":"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b","Type":"ContainerDied","Data":"4d6ee4231caf0ff6195e9a7a283a64dbf8701386be08bebd449402ec8edd2532"} Sep 16 13:47:42.721377 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:42.721332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" event={"ID":"4a92a250-a4cc-4c2f-98ca-c5aa03acc32b","Type":"ContainerStarted","Data":"d9ba829ccead65a5d1822cd31d28a83280da24e0ca198ca2adf6a665a80eb108"} Sep 16 13:47:42.721765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:42.721459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:42.744782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:42.744727 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" podStartSLOduration=2.411202507 podStartE2EDuration="6.744709957s" totalImagesPullingTime="3.289456907s" totalInitContainerRuntime="1.044050543s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:42.742084938 +0000 UTC m=+281.782678893" watchObservedRunningTime="2026-09-16 13:47:42.744709957 +0000 UTC m=+281.785303912" Sep 16 13:47:47.607055 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:47.607014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:48.315807 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:48.315768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:53.728247 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:53.728212 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-7sfbx" Sep 16 13:47:55.630017 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:55.629969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:57.493364 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:47:57.493335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:01.398049 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:01.398021 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 13:48:06.300430 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.300370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:06.325153 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.325116 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s"] Sep 16 13:48:06.329199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.329174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.331638 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.331619 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 13:48:06.332018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.331993 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n25nr\"" Sep 16 13:48:06.332152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.332012 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 13:48:06.332152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.332075 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 13:48:06.332736 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.332714 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 13:48:06.342970 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.342946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s"] Sep 16 13:48:06.460314 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.460263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswgq\" (UniqueName: \"kubernetes.io/projected/e3500fa6-aac0-4404-8acb-e0452a8e6a01-kube-api-access-fswgq\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.460480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.460321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.460480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.460456 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.561487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.561387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.561487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.561449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswgq\" (UniqueName: \"kubernetes.io/projected/e3500fa6-aac0-4404-8acb-e0452a8e6a01-kube-api-access-fswgq\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.561487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.561478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.561818 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:48:06.561608 2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 16 13:48:06.561818 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:48:06.561689 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs podName:e3500fa6-aac0-4404-8acb-e0452a8e6a01 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:07.061668059 +0000 UTC m=+306.102261994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs") pod "kube-auth-proxy-66fc9bc5dc-j8p2s" (UID: "e3500fa6-aac0-4404-8acb-e0452a8e6a01") : secret "kube-auth-proxy-tls" not found Sep 16 13:48:06.563673 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.563650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.573181 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.573152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswgq\" (UniqueName: \"kubernetes.io/projected/e3500fa6-aac0-4404-8acb-e0452a8e6a01-kube-api-access-fswgq\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:06.642200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.642165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:06.941311 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.941228 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h9d4"] Sep 16 13:48:06.945417 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.945398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:06.948437 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.948411 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 16 13:48:06.958533 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.958502 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jfgbp\"" Sep 16 13:48:06.959276 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.959253 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h9d4"] Sep 16 13:48:06.988860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:06.988820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.067148 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.067098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgg6\" (UniqueName: \"kubernetes.io/projected/93d4bf31-20f4-4f5d-ac8e-f4378f782545-kube-api-access-fsgg6\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.067321 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.067247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.067374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.067319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:07.070246 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.070214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3500fa6-aac0-4404-8acb-e0452a8e6a01-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-j8p2s\" (UID: \"e3500fa6-aac0-4404-8acb-e0452a8e6a01\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:07.099881 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.099832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.168290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.168252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.168505 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.168363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgg6\" (UniqueName: \"kubernetes.io/projected/93d4bf31-20f4-4f5d-ac8e-f4378f782545-kube-api-access-fsgg6\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.168505 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:48:07.168426 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 16 13:48:07.168505 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:48:07.168499 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert podName:93d4bf31-20f4-4f5d-ac8e-f4378f782545 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:07.668478222 +0000 UTC m=+306.709072160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert") pod "odh-model-controller-7998c67dc4-8h9d4" (UID: "93d4bf31-20f4-4f5d-ac8e-f4378f782545") : secret "odh-model-controller-webhook-cert" not found Sep 16 13:48:07.188885 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.188842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgg6\" (UniqueName: \"kubernetes.io/projected/93d4bf31-20f4-4f5d-ac8e-f4378f782545-kube-api-access-fsgg6\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.208224 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.208194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.239250 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.239216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" Sep 16 13:48:07.503488 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.503407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.516443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.516411 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s"] Sep 16 13:48:07.519193 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:48:07.519166 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3500fa6_aac0_4404_8acb_e0452a8e6a01.slice/crio-c33b7e75d7787a00103a76cba30fbbd1eb73047751109a5ccf1f0d0000ee59bb WatchSource:0}: Error finding container c33b7e75d7787a00103a76cba30fbbd1eb73047751109a5ccf1f0d0000ee59bb: Status 404 returned error can't find the container with id c33b7e75d7787a00103a76cba30fbbd1eb73047751109a5ccf1f0d0000ee59bb Sep 16 13:48:07.521061 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.521042 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:48:07.639332 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.639302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.667175 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.667141 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:07.672983 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.672955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.673135 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.673113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.675490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675456 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 13:48:07.675672 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675516 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cwdfv\"" Sep 16 13:48:07.675744 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93d4bf31-20f4-4f5d-ac8e-f4378f782545-cert\") pod \"odh-model-controller-7998c67dc4-8h9d4\" (UID: \"93d4bf31-20f4-4f5d-ac8e-f4378f782545\") " pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.675924 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675906 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 13:48:07.676010 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675937 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:48:07.676010 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.675937 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 13:48:07.686363 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.686338 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:07.744186 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.744141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.773603 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773503 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773603 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5n57\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773603 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773603 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773602 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.773887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.773696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.816597 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.816561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:07.827362 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.827329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" event={"ID":"e3500fa6-aac0-4404-8acb-e0452a8e6a01","Type":"ContainerStarted","Data":"c33b7e75d7787a00103a76cba30fbbd1eb73047751109a5ccf1f0d0000ee59bb"} Sep 16 13:48:07.859796 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.859763 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:07.875247 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5n57\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.875558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.875505 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.876417 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.876380 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.878384 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.878351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.878488 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.878407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.879078 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.879055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.880397 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.880372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.884702 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.884676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.885033 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.885010 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5n57\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57\") pod \"istiod-openshift-gateway-7b7995d5cd-cz468\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:07.997535 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:07.997496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:08.074746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.074717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:08.135814 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.135788 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8h9d4"] Sep 16 13:48:08.137756 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:48:08.137728 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d4bf31_20f4_4f5d_ac8e_f4378f782545.slice/crio-7cd793ca116bfa8537e8ea20f1a47275751ec8f6c8c89b35ae0591a8eb32eae5 WatchSource:0}: Error finding container 7cd793ca116bfa8537e8ea20f1a47275751ec8f6c8c89b35ae0591a8eb32eae5: Status 404 returned error can't find the container with id 7cd793ca116bfa8537e8ea20f1a47275751ec8f6c8c89b35ae0591a8eb32eae5 Sep 16 13:48:08.297466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.297426 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:08.299937 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:48:08.299911 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a19a61_6792_4a80_a37d_704189062b2b.slice/crio-ddaefc5d5bb58c01f264a313f8228c951683fa8666dedf948deb48f400aea121 WatchSource:0}: Error finding container ddaefc5d5bb58c01f264a313f8228c951683fa8666dedf948deb48f400aea121: Status 404 returned error can't find the container with id ddaefc5d5bb58c01f264a313f8228c951683fa8666dedf948deb48f400aea121 Sep 16 13:48:08.831409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.831372 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:08.834682 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.834646 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" event={"ID":"93d4bf31-20f4-4f5d-ac8e-f4378f782545","Type":"ContainerStarted","Data":"7cd793ca116bfa8537e8ea20f1a47275751ec8f6c8c89b35ae0591a8eb32eae5"} Sep 16 13:48:08.836170 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:08.836143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" event={"ID":"f6a19a61-6792-4a80-a37d-704189062b2b","Type":"ContainerStarted","Data":"ddaefc5d5bb58c01f264a313f8228c951683fa8666dedf948deb48f400aea121"} Sep 16 13:48:12.865237 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:12.865189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" event={"ID":"e3500fa6-aac0-4404-8acb-e0452a8e6a01","Type":"ContainerStarted","Data":"7e5546627b54fd7ff41da3619bb5f7fbc6f4dac7ab794aa5f0ccfcdd54691987"} Sep 16 13:48:12.868406 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:12.868148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" event={"ID":"93d4bf31-20f4-4f5d-ac8e-f4378f782545","Type":"ContainerStarted","Data":"e71a4ee55ffb1aeea0690eb9110ba46daed5e78f4b0f92bc7f4b9e092e13bc85"} Sep 16 13:48:12.868568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:12.868477 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:12.887927 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:12.887847 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-j8p2s" podStartSLOduration=1.998611634 podStartE2EDuration="6.887824387s" totalImagesPullingTime="4.889212753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:12.885914661 +0000 UTC m=+311.926508615" watchObservedRunningTime="2026-09-16 13:48:12.887824387 +0000 UTC m=+311.928418343" Sep 16 13:48:14.356838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:14.356797 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:48:14.357150 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:14.356881 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:48:14.885837 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:14.885789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" event={"ID":"f6a19a61-6792-4a80-a37d-704189062b2b","Type":"ContainerStarted","Data":"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006"} Sep 16 13:48:14.886044 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:14.885992 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:14.926646 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:14.926593 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" podStartSLOduration=4.613738532 podStartE2EDuration="8.926577048s" totalImagesPullingTime="4.312838516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:12.905483434 +0000 UTC m=+311.946077405" watchObservedRunningTime="2026-09-16 13:48:14.926577048 +0000 UTC m=+313.967171003" Sep 16 13:48:15.484197 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:15.484159 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:15.605962 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:15.605919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:15.892070 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:15.891986 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cz468 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:48:15.892070 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:15.892056 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:48:16.436331 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.436274 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" podStartSLOduration=3.381673991 podStartE2EDuration="9.436254319s" totalImagesPullingTime="6.054580328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:14.928569574 +0000 UTC m=+313.969163541" watchObservedRunningTime="2026-09-16 13:48:16.436254319 +0000 UTC m=+315.476848274" Sep 16 13:48:16.436854 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.436839 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz"] Sep 16 13:48:16.440465 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.440444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.454335 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.454306 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:48:16.455249 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.455226 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 13:48:16.455369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.455243 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 13:48:16.455369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.455263 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 13:48:16.455369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.455332 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7kb6t\"" Sep 16 13:48:16.455488 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.455437 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 13:48:16.464068 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.464043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.464167 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.464089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.464226 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.464158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6rn\" (UniqueName: \"kubernetes.io/projected/c8ffc9fe-8377-4047-912c-bedca466d013-kube-api-access-fl6rn\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.464226 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.464217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8ffc9fe-8377-4047-912c-bedca466d013-manager-config\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.466691 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.466667 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz"] Sep 16 13:48:16.564703 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.564664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8ffc9fe-8377-4047-912c-bedca466d013-manager-config\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.565255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.565031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.565255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.565093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.565255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.565162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6rn\" (UniqueName: \"kubernetes.io/projected/c8ffc9fe-8377-4047-912c-bedca466d013-kube-api-access-fl6rn\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.565482 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.565453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8ffc9fe-8377-4047-912c-bedca466d013-manager-config\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.567457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.567436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.567560 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.567487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8ffc9fe-8377-4047-912c-bedca466d013-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.585938 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.585903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6rn\" (UniqueName: \"kubernetes.io/projected/c8ffc9fe-8377-4047-912c-bedca466d013-kube-api-access-fl6rn\") pod \"lws-controller-manager-6465fbddb8-rn9vz\" (UID: \"c8ffc9fe-8377-4047-912c-bedca466d013\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.735937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.735900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:16.749104 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.749063 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:16.853206 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:16.853170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:17.007937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:17.007834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:17.035031 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:17.034999 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz"] Sep 16 13:48:17.037925 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:48:17.037901 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ffc9fe_8377_4047_912c_bedca466d013.slice/crio-19ac665e622ac5d61dfa83a3264ea0077db044125aa5f1237c8f1e7dfecea3e4 WatchSource:0}: Error finding container 19ac665e622ac5d61dfa83a3264ea0077db044125aa5f1237c8f1e7dfecea3e4: Status 404 returned error can't find the container with id 19ac665e622ac5d61dfa83a3264ea0077db044125aa5f1237c8f1e7dfecea3e4 Sep 16 13:48:17.903133 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:17.903094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" event={"ID":"c8ffc9fe-8377-4047-912c-bedca466d013","Type":"ContainerStarted","Data":"19ac665e622ac5d61dfa83a3264ea0077db044125aa5f1237c8f1e7dfecea3e4"} Sep 16 13:48:18.892051 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:18.892018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:19.249972 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.249921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:19.359533 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.359496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:19.587036 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.587011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:19.914426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.914337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" event={"ID":"c8ffc9fe-8377-4047-912c-bedca466d013","Type":"ContainerStarted","Data":"cae7eae2efb751e3c4027bcaaa30da7b046d7cc1637f94493779bc896f44719f"} Sep 16 13:48:19.914588 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.914457 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:48:19.933774 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:19.933718 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" podStartSLOduration=1.4118686249999999 podStartE2EDuration="3.933700923s" totalImagesPullingTime="2.521832298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:19.932639243 +0000 UTC m=+318.973233199" watchObservedRunningTime="2026-09-16 13:48:19.933700923 +0000 UTC m=+318.974294878" Sep 16 13:48:20.202004 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.201965 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:20.202172 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.202149 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" containerName="discovery" containerID="cri-o://d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006" gracePeriod=30 Sep 16 13:48:20.409310 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.409278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:20.436368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.436341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:20.499940 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.499842 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.499940 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.499918 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500017 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500072 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5n57\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500179 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert\") pod \"f6a19a61-6792-4a80-a37d-704189062b2b\" (UID: \"f6a19a61-6792-4a80-a37d-704189062b2b\") " Sep 16 13:48:20.500323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500256 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:48:20.500554 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.500533 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-ca-configmap\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.502523 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502498 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:48:20.502657 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502626 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts" (OuterVolumeSpecName: "cacerts") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:48:20.502715 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502661 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:48:20.502715 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs" (OuterVolumeSpecName: "local-certs") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:48:20.502806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502751 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57" (OuterVolumeSpecName: "kube-api-access-f5n57") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "kube-api-access-f5n57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:48:20.502856 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.502818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f6a19a61-6792-4a80-a37d-704189062b2b" (UID: "f6a19a61-6792-4a80-a37d-704189062b2b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:48:20.550583 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.550551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:20.601520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601483 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5n57\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-kube-api-access-f5n57\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.601520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601516 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a19a61-6792-4a80-a37d-704189062b2b-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.601520 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601525 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-cacerts\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.601771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601533 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-csr-dns-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.601771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601542 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a19a61-6792-4a80-a37d-704189062b2b-local-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.601771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.601550 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a19a61-6792-4a80-a37d-704189062b2b-istio-kubeconfig\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:48:20.653668 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.653630 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:20.751533 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.751449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:20.920239 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.920206 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6a19a61-6792-4a80-a37d-704189062b2b" containerID="d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006" exitCode=0 Sep 16 13:48:20.920461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.920254 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" Sep 16 13:48:20.920461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.920297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" event={"ID":"f6a19a61-6792-4a80-a37d-704189062b2b","Type":"ContainerDied","Data":"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006"} Sep 16 13:48:20.920461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.920345 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468" event={"ID":"f6a19a61-6792-4a80-a37d-704189062b2b","Type":"ContainerDied","Data":"ddaefc5d5bb58c01f264a313f8228c951683fa8666dedf948deb48f400aea121"} Sep 16 13:48:20.920461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.920368 2559 scope.go:117] "RemoveContainer" containerID="d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006" Sep 16 13:48:20.929032 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.929011 2559 scope.go:117] "RemoveContainer" containerID="d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006" Sep 16 13:48:20.929306 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:48:20.929289 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006\": container with ID starting with d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006 not found: ID does not exist" containerID="d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006" Sep 16 13:48:20.929368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.929313 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006"} err="failed to get container status \"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006\": rpc error: code = NotFound desc = could not find container \"d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006\": container with ID starting with d2101b40239bd876a45c5621604c6d65969cc6e6655f1f87fa60b69b7511c006 not found: ID does not exist" Sep 16 13:48:20.951207 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.951177 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:20.954782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:20.954751 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cz468"] Sep 16 13:48:21.472490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:21.472459 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" path="/var/lib/kubelet/pods/f6a19a61-6792-4a80-a37d-704189062b2b/volumes" Sep 16 13:48:22.467365 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:22.467328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:22.566583 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:22.566541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:23.876291 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:23.876252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8h9d4" Sep 16 13:48:30.923031 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:48:30.922999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-rn9vz" Sep 16 13:49:08.013849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:08.013810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:08.111222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:08.111181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:27.530447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:27.530413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:31.167666 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:31.167631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:31.737437 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:31.737400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:31.874853 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:31.874822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:31.975775 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:31.975734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:32.467615 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:32.467581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:32.576539 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:32.576501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:33.514754 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:33.514727 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:34.143637 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:34.143600 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:36.248485 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:36.248447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:36.400284 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:36.400243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:36.767472 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:36.767444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:37.169399 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:37.169317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:37.411965 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:37.411934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:37.572307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:37.572276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:38.931605 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:38.931574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:39.611152 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:39.611118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:39.760612 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:39.760571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:42.279338 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:42.279307 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:42.413274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:42.413230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:42.800141 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:42.800104 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:42.900718 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:42.900682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:44.939981 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:44.939929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:45.046623 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:45.046584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:47.788574 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.788541 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr"] Sep 16 13:49:47.788971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.788893 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" containerName="discovery" Sep 16 13:49:47.788971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.788906 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" containerName="discovery" Sep 16 13:49:47.788971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.788957 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a19a61-6792-4a80-a37d-704189062b2b" containerName="discovery" Sep 16 13:49:47.792102 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.792082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:47.792911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.792885 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:47.794584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.794565 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 13:49:47.794671 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.794596 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 13:49:47.795300 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.795284 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vfw4f\"" Sep 16 13:49:47.795370 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.795295 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 13:49:47.795370 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.795298 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 13:49:47.805864 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.805843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr"] Sep 16 13:49:47.918740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.918705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:47.918917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.918752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:47.918917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:47.918788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vqd\" (UniqueName: \"kubernetes.io/projected/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-kube-api-access-g7vqd\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.005471 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.005435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:48.019577 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.019549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.019724 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.019590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.019724 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.019638 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vqd\" (UniqueName: \"kubernetes.io/projected/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-kube-api-access-g7vqd\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.020380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.020359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.021999 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.021974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.029497 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.029473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vqd\" (UniqueName: \"kubernetes.io/projected/e3d44aa3-6f60-45ce-ac22-80726fd6aa59-kube-api-access-g7vqd\") pod \"kuadrant-console-plugin-fbdd4f445-n44pr\" (UID: \"e3d44aa3-6f60-45ce-ac22-80726fd6aa59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.101092 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.101002 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" Sep 16 13:49:48.373773 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:48.373746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr"] Sep 16 13:49:48.376245 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:49:48.376219 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d44aa3_6f60_45ce_ac22_80726fd6aa59.slice/crio-2a3d65d682a746440465d0f530bc5aa1f2e86b47bbd3b86977628e072b7b65aa WatchSource:0}: Error finding container 2a3d65d682a746440465d0f530bc5aa1f2e86b47bbd3b86977628e072b7b65aa: Status 404 returned error can't find the container with id 2a3d65d682a746440465d0f530bc5aa1f2e86b47bbd3b86977628e072b7b65aa Sep 16 13:49:49.313344 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:49.313294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" event={"ID":"e3d44aa3-6f60-45ce-ac22-80726fd6aa59","Type":"ContainerStarted","Data":"2a3d65d682a746440465d0f530bc5aa1f2e86b47bbd3b86977628e072b7b65aa"} Sep 16 13:49:49.951827 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:49.951787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:50.053353 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:50.053306 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:53.335143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:53.335108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" event={"ID":"e3d44aa3-6f60-45ce-ac22-80726fd6aa59","Type":"ContainerStarted","Data":"0d4b8c037229b06966741cd329f481606614abb231d8ff6c48e63bb5f22e4df6"} Sep 16 13:49:53.352802 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:53.352752 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n44pr" podStartSLOduration=2.163960598 podStartE2EDuration="6.352737076s" totalImagesPullingTime="4.188776478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:53.351429535 +0000 UTC m=+412.392023493" watchObservedRunningTime="2026-09-16 13:49:53.352737076 +0000 UTC m=+412.393331031" Sep 16 13:49:54.675836 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:54.675805 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:54.776562 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:54.776530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:54.882022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:54.881991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:59.565182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:59.565152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:59.664924 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:49:59.664888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:01.526472 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:01.526438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:01.626826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:01.626790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:04.149297 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:04.149261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:04.259411 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:04.259375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:09.148993 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:09.148956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:09.249398 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:09.249367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:13.919501 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:13.919465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:14.020126 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:14.020088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:14.126018 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:14.125985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:30.479551 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:30.479468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:30.539841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:30.539811 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:43.961584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:43.961549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:44.063359 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.063323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:44.166863 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.166827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:44.494769 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.494733 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:50:44.498083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.498057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.500049 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.500030 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 13:50:44.509644 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.509622 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:50:44.552359 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.552328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:50:44.598274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.598236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbqk\" (UniqueName: \"kubernetes.io/projected/007c82fb-d49e-4171-8d3c-4a15edac954d-kube-api-access-vlbqk\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.598438 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.598286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/007c82fb-d49e-4171-8d3c-4a15edac954d-config-file\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.699346 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.699312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbqk\" (UniqueName: \"kubernetes.io/projected/007c82fb-d49e-4171-8d3c-4a15edac954d-kube-api-access-vlbqk\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.699512 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.699364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/007c82fb-d49e-4171-8d3c-4a15edac954d-config-file\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.699977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.699959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/007c82fb-d49e-4171-8d3c-4a15edac954d-config-file\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.708060 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.708023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbqk\" (UniqueName: \"kubernetes.io/projected/007c82fb-d49e-4171-8d3c-4a15edac954d-kube-api-access-vlbqk\") pod \"limitador-limitador-84994f7ffc-8msjx\" (UID: \"007c82fb-d49e-4171-8d3c-4a15edac954d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.807891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.807777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:44.859424 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.859395 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:44.863213 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.863192 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:44.865147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.865127 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jqwlz\"" Sep 16 13:50:44.875406 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.875350 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:44.900796 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.900762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d7k\" (UniqueName: \"kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k\") pod \"authorino-7bcfd8f9cd-lrbgt\" (UID: \"b1147aca-39e0-4172-8e8d-756e064e5909\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:44.902570 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:44.902533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:45.002168 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.002134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d7k\" (UniqueName: \"kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k\") pod \"authorino-7bcfd8f9cd-lrbgt\" (UID: \"b1147aca-39e0-4172-8e8d-756e064e5909\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:45.006210 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.006187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:45.013484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.013460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d7k\" (UniqueName: \"kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k\") pod \"authorino-7bcfd8f9cd-lrbgt\" (UID: \"b1147aca-39e0-4172-8e8d-756e064e5909\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:45.090340 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.090265 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:50:45.093571 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:50:45.093547 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007c82fb_d49e_4171_8d3c_4a15edac954d.slice/crio-7c0467e8bff09c1287ccee06a4c727845c9d07ace0e1fef8bb4e0df483dcb2f1 WatchSource:0}: Error finding container 7c0467e8bff09c1287ccee06a4c727845c9d07ace0e1fef8bb4e0df483dcb2f1: Status 404 returned error can't find the container with id 7c0467e8bff09c1287ccee06a4c727845c9d07ace0e1fef8bb4e0df483dcb2f1 Sep 16 13:50:45.177485 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.177448 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:45.445080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.445047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:45.448231 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:50:45.448204 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1147aca_39e0_4172_8e8d_756e064e5909.slice/crio-b6b3e69d0d702df77023be582e5e52331280a6402d880733a377531ce647a93c WatchSource:0}: Error finding container b6b3e69d0d702df77023be582e5e52331280a6402d880733a377531ce647a93c: Status 404 returned error can't find the container with id b6b3e69d0d702df77023be582e5e52331280a6402d880733a377531ce647a93c Sep 16 13:50:45.556047 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.556014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" event={"ID":"007c82fb-d49e-4171-8d3c-4a15edac954d","Type":"ContainerStarted","Data":"7c0467e8bff09c1287ccee06a4c727845c9d07ace0e1fef8bb4e0df483dcb2f1"} Sep 16 13:50:45.557180 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:45.557157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" event={"ID":"b1147aca-39e0-4172-8e8d-756e064e5909","Type":"ContainerStarted","Data":"b6b3e69d0d702df77023be582e5e52331280a6402d880733a377531ce647a93c"} Sep 16 13:50:47.933686 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:47.933653 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:48.580784 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.580665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" event={"ID":"007c82fb-d49e-4171-8d3c-4a15edac954d","Type":"ContainerStarted","Data":"a8b1476d5815a1556b8120a4fe70acac8ff394d7adba4d655aabd66395d0bcdd"} Sep 16 13:50:48.580985 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.580781 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:50:48.582187 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.582161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" event={"ID":"b1147aca-39e0-4172-8e8d-756e064e5909","Type":"ContainerStarted","Data":"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974"} Sep 16 13:50:48.582304 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.582225 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" podUID="b1147aca-39e0-4172-8e8d-756e064e5909" containerName="authorino" containerID="cri-o://32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974" gracePeriod=30 Sep 16 13:50:48.600269 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.600195 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" podStartSLOduration=1.376677926 podStartE2EDuration="4.600174256s" totalImagesPullingTime="3.22349633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:48.598726168 +0000 UTC m=+467.639320126" watchObservedRunningTime="2026-09-16 13:50:48.600174256 +0000 UTC m=+467.640768213" Sep 16 13:50:48.613806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.613756 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" podStartSLOduration=1.7382909450000001 podStartE2EDuration="4.61374091s" totalImagesPullingTime="2.875449965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:48.61351818 +0000 UTC m=+467.654112134" watchObservedRunningTime="2026-09-16 13:50:48.61374091 +0000 UTC m=+467.654334865" Sep 16 13:50:48.833710 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.833641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:48.928356 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.928316 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5d7k\" (UniqueName: \"kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k\") pod \"b1147aca-39e0-4172-8e8d-756e064e5909\" (UID: \"b1147aca-39e0-4172-8e8d-756e064e5909\") " Sep 16 13:50:48.930473 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:48.930449 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k" (OuterVolumeSpecName: "kube-api-access-r5d7k") pod "b1147aca-39e0-4172-8e8d-756e064e5909" (UID: "b1147aca-39e0-4172-8e8d-756e064e5909"). InnerVolumeSpecName "kube-api-access-r5d7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:49.029581 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.029537 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d7k\" (UniqueName: \"kubernetes.io/projected/b1147aca-39e0-4172-8e8d-756e064e5909-kube-api-access-r5d7k\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:50:49.588828 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.588729 2559 generic.go:356] "Generic (PLEG): container finished" podID="b1147aca-39e0-4172-8e8d-756e064e5909" containerID="32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974" exitCode=0 Sep 16 13:50:49.588828 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.588770 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" Sep 16 13:50:49.588828 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.588810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" event={"ID":"b1147aca-39e0-4172-8e8d-756e064e5909","Type":"ContainerDied","Data":"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974"} Sep 16 13:50:49.589102 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.588840 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lrbgt" event={"ID":"b1147aca-39e0-4172-8e8d-756e064e5909","Type":"ContainerDied","Data":"b6b3e69d0d702df77023be582e5e52331280a6402d880733a377531ce647a93c"} Sep 16 13:50:49.589102 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.588857 2559 scope.go:117] "RemoveContainer" containerID="32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974" Sep 16 13:50:49.597431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.597404 2559 scope.go:117] "RemoveContainer" containerID="32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974" Sep 16 13:50:49.597789 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:50:49.597767 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974\": container with ID starting with 32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974 not found: ID does not exist" containerID="32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974" Sep 16 13:50:49.597997 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.597796 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974"} err="failed to get container status \"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974\": rpc error: code = NotFound desc = could not find container \"32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974\": container with ID starting with 32fbe687ac04738539c29c69e7ac20dcc60543a166a4beb4281c0c582f9d2974 not found: ID does not exist" Sep 16 13:50:49.608403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.608366 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:49.612258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:49.612230 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lrbgt"] Sep 16 13:50:51.471017 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:51.470981 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1147aca-39e0-4172-8e8d-756e064e5909" path="/var/lib/kubelet/pods/b1147aca-39e0-4172-8e8d-756e064e5909/volumes" Sep 16 13:50:59.589998 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:50:59.589965 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8msjx" Sep 16 13:51:04.393735 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:04.393695 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:04.499895 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:04.499840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:04.884968 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:04.884930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:09.392958 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:09.392923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.484674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.484641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.712644 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.712607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:17.712984 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.712970 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1147aca-39e0-4172-8e8d-756e064e5909" containerName="authorino" Sep 16 13:51:17.713035 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.712986 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1147aca-39e0-4172-8e8d-756e064e5909" containerName="authorino" Sep 16 13:51:17.713070 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.713042 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1147aca-39e0-4172-8e8d-756e064e5909" containerName="authorino" Sep 16 13:51:17.717909 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.717892 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:17.720127 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.720106 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jqwlz\"" Sep 16 13:51:17.727312 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.727283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:17.767844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.767755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hmw\" (UniqueName: \"kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw\") pod \"authorino-557f876f69-zzw57\" (UID: \"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b\") " pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:17.869041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.869006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hmw\" (UniqueName: \"kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw\") pod \"authorino-557f876f69-zzw57\" (UID: \"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b\") " pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:17.878219 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.878194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hmw\" (UniqueName: \"kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw\") pod \"authorino-557f876f69-zzw57\" (UID: \"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b\") " pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:17.883064 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.883037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.965575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.965535 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:17.965818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.965806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:17.995515 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.995481 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58c68c9d84-f4jrb"] Sep 16 13:51:17.998957 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:17.998934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.011238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.011213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58c68c9d84-f4jrb"] Sep 16 13:51:18.054730 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.054692 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c68c9d84-f4jrb"] Sep 16 13:51:18.055027 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:51:18.055003 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pl8lf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-58c68c9d84-f4jrb" podUID="320d56f1-c8be-49a8-90c6-52d7736a12db" Sep 16 13:51:18.070175 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.070145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8lf\" (UniqueName: \"kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf\") pod \"authorino-58c68c9d84-f4jrb\" (UID: \"320d56f1-c8be-49a8-90c6-52d7736a12db\") " pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.097747 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.097712 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:51:18.100597 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.100580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.102569 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.102548 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 13:51:18.108397 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.108376 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:51:18.171480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.171441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.171480 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.171480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rc2l\" (UniqueName: \"kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.171716 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.171509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8lf\" (UniqueName: \"kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf\") pod \"authorino-58c68c9d84-f4jrb\" (UID: \"320d56f1-c8be-49a8-90c6-52d7736a12db\") " pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.181419 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.181393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8lf\" (UniqueName: \"kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf\") pod \"authorino-58c68c9d84-f4jrb\" (UID: \"320d56f1-c8be-49a8-90c6-52d7736a12db\") " pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.237920 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.237772 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:18.240688 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:51:18.240660 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6e64f10_8c68_4afb_93eb_47f6cbb9d29b.slice/crio-fe6e864a658be76c2e3095f4968b39d6ed62afc578fc139eac13be7146219052 WatchSource:0}: Error finding container fe6e864a658be76c2e3095f4968b39d6ed62afc578fc139eac13be7146219052: Status 404 returned error can't find the container with id fe6e864a658be76c2e3095f4968b39d6ed62afc578fc139eac13be7146219052 Sep 16 13:51:18.273059 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.272961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.273059 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.273009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rc2l\" (UniqueName: \"kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.275489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.275462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.291720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.291685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rc2l\" (UniqueName: \"kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l\") pod \"authorino-7d4d9f959d-r4sdd\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.409861 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.409812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:51:18.680711 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.680666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:51:18.682853 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:51:18.682823 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fb934be_df2d_46c6_a73b_4d9e936282ee.slice/crio-90ac1c80915a22cd7f4f45c65aa7c913b60c157ad92a87d9d3bad16b20b97e15 WatchSource:0}: Error finding container 90ac1c80915a22cd7f4f45c65aa7c913b60c157ad92a87d9d3bad16b20b97e15: Status 404 returned error can't find the container with id 90ac1c80915a22cd7f4f45c65aa7c913b60c157ad92a87d9d3bad16b20b97e15 Sep 16 13:51:18.717323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.717291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" event={"ID":"9fb934be-df2d-46c6-a73b-4d9e936282ee","Type":"ContainerStarted","Data":"90ac1c80915a22cd7f4f45c65aa7c913b60c157ad92a87d9d3bad16b20b97e15"} Sep 16 13:51:18.718478 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.718456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.718596 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.718484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zzw57" event={"ID":"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b","Type":"ContainerStarted","Data":"fe6e864a658be76c2e3095f4968b39d6ed62afc578fc139eac13be7146219052"} Sep 16 13:51:18.726342 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.726321 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:18.777796 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.777759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl8lf\" (UniqueName: \"kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf\") pod \"320d56f1-c8be-49a8-90c6-52d7736a12db\" (UID: \"320d56f1-c8be-49a8-90c6-52d7736a12db\") " Sep 16 13:51:18.779809 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.779781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf" (OuterVolumeSpecName: "kube-api-access-pl8lf") pod "320d56f1-c8be-49a8-90c6-52d7736a12db" (UID: "320d56f1-c8be-49a8-90c6-52d7736a12db"). InnerVolumeSpecName "kube-api-access-pl8lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:18.879258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:18.879224 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl8lf\" (UniqueName: \"kubernetes.io/projected/320d56f1-c8be-49a8-90c6-52d7736a12db-kube-api-access-pl8lf\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:51:19.724304 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.724263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" event={"ID":"9fb934be-df2d-46c6-a73b-4d9e936282ee","Type":"ContainerStarted","Data":"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8"} Sep 16 13:51:19.725834 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.725768 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c68c9d84-f4jrb" Sep 16 13:51:19.725834 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.725783 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zzw57" event={"ID":"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b","Type":"ContainerStarted","Data":"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41"} Sep 16 13:51:19.726008 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.725864 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-zzw57" podUID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" containerName="authorino" containerID="cri-o://8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41" gracePeriod=30 Sep 16 13:51:19.739897 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.739841 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" podStartSLOduration=1.242028695 podStartE2EDuration="1.739825637s" totalImagesPullingTime="497.796942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:19.739016215 +0000 UTC m=+498.779610184" watchObservedRunningTime="2026-09-16 13:51:19.739825637 +0000 UTC m=+498.780419592" Sep 16 13:51:19.763095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.763063 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c68c9d84-f4jrb"] Sep 16 13:51:19.773198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.773168 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58c68c9d84-f4jrb"] Sep 16 13:51:19.790231 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.790180 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-zzw57" podStartSLOduration=2.237260098 podStartE2EDuration="2.790164169s" totalImagesPullingTime="552.904071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:19.789285524 +0000 UTC m=+498.829879479" watchObservedRunningTime="2026-09-16 13:51:19.790164169 +0000 UTC m=+498.830758125" Sep 16 13:51:19.969175 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:19.969151 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:20.091842 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.091742 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9hmw\" (UniqueName: \"kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw\") pod \"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b\" (UID: \"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b\") " Sep 16 13:51:20.093888 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.093833 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw" (OuterVolumeSpecName: "kube-api-access-f9hmw") pod "b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" (UID: "b6e64f10-8c68-4afb-93eb-47f6cbb9d29b"). InnerVolumeSpecName "kube-api-access-f9hmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:20.193138 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.193101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hmw\" (UniqueName: \"kubernetes.io/projected/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b-kube-api-access-f9hmw\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:51:20.731516 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.731481 2559 generic.go:356] "Generic (PLEG): container finished" podID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" containerID="8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41" exitCode=0 Sep 16 13:51:20.731516 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.731516 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zzw57" Sep 16 13:51:20.732065 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.731560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zzw57" event={"ID":"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b","Type":"ContainerDied","Data":"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41"} Sep 16 13:51:20.732065 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.731598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-zzw57" event={"ID":"b6e64f10-8c68-4afb-93eb-47f6cbb9d29b","Type":"ContainerDied","Data":"fe6e864a658be76c2e3095f4968b39d6ed62afc578fc139eac13be7146219052"} Sep 16 13:51:20.732065 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.731614 2559 scope.go:117] "RemoveContainer" containerID="8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41" Sep 16 13:51:20.740777 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.740760 2559 scope.go:117] "RemoveContainer" containerID="8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41" Sep 16 13:51:20.741065 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:51:20.741049 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41\": container with ID starting with 8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41 not found: ID does not exist" containerID="8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41" Sep 16 13:51:20.741122 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.741074 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41"} err="failed to get container status \"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41\": rpc error: code = NotFound desc = could not find container \"8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41\": container with ID starting with 8d780b2b1c064f7191bac98265fe58ab43792b2059129d58a98cddc162a9df41 not found: ID does not exist" Sep 16 13:51:20.757008 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.756985 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:20.762968 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:20.762934 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zzw57"] Sep 16 13:51:21.472600 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:21.472565 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320d56f1-c8be-49a8-90c6-52d7736a12db" path="/var/lib/kubelet/pods/320d56f1-c8be-49a8-90c6-52d7736a12db/volumes" Sep 16 13:51:21.472795 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:21.472783 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" path="/var/lib/kubelet/pods/b6e64f10-8c68-4afb-93eb-47f6cbb9d29b/volumes" Sep 16 13:51:24.732219 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:24.732190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:24.854140 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:24.854103 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:24.950579 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:24.950538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:25.047090 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:25.046993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:27.489931 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:27.489894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:28.294068 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:28.294032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:28.394852 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:28.394816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:36.061191 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:51:36.061153 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:00.779714 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:00.779636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:00.877564 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:00.877523 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:02.352707 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:02.352672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:02.458718 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:02.458680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:02.553736 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:02.553703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:03.010362 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:03.010325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:03.109135 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:03.109098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:03.201287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:03.201250 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:52:05.692436 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:05.692389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:12.462454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:12.462363 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:12.562004 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:12.561967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:32.368527 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:32.368486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:32.845566 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:32.845524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:52:37.461135 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:37.461093 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:38.071940 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:38.071905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:52:43.461922 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:43.461888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:44.348440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:52:44.348399 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:53:06.681083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:06.681050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:07.942182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:07.942146 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:53:16.872161 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:16.872127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:18.535735 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:18.535695 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:53:59.383117 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:59.383077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:59.447227 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:53:59.447195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:02.159978 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.159936 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg"] Sep 16 13:54:02.160486 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.160465 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" containerName="authorino" Sep 16 13:54:02.160561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.160490 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" containerName="authorino" Sep 16 13:54:02.160617 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.160569 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e64f10-8c68-4afb-93eb-47f6cbb9d29b" containerName="authorino" Sep 16 13:54:02.163519 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.163498 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.166108 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.166084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 13:54:02.166226 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.166195 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g67b7\"" Sep 16 13:54:02.166660 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.166644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 13:54:02.166818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.166801 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 16 13:54:02.176743 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.176718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg"] Sep 16 13:54:02.260854 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.260811 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:02.297920 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.297859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxk5v\" (UniqueName: \"kubernetes.io/projected/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-kube-api-access-fxk5v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.297920 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.297917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.298176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.297939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.298176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.297991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.298176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.298016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.298176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.298037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399501 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxk5v\" (UniqueName: \"kubernetes.io/projected/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-kube-api-access-fxk5v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399820 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.399820 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.399734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.400027 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.400004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.400094 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.400076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.400182 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.400161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.402029 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.402009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.402217 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.402199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.410315 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.410259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxk5v\" (UniqueName: \"kubernetes.io/projected/f8d07bb8-f953-4e1d-abce-ee0c89995cd3-kube-api-access-fxk5v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbdpg\" (UID: \"f8d07bb8-f953-4e1d-abce-ee0c89995cd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.478699 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.478660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:02.754280 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.754257 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg"] Sep 16 13:54:02.756383 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:54:02.756355 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d07bb8_f953_4e1d_abce_ee0c89995cd3.slice/crio-80a87d733e68684cf41a62834cc2fe1e71599939cdad84e06433529d8387dd99 WatchSource:0}: Error finding container 80a87d733e68684cf41a62834cc2fe1e71599939cdad84e06433529d8387dd99: Status 404 returned error can't find the container with id 80a87d733e68684cf41a62834cc2fe1e71599939cdad84e06433529d8387dd99 Sep 16 13:54:02.757625 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:02.757607 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:54:03.453052 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:03.453007 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" event={"ID":"f8d07bb8-f953-4e1d-abce-ee0c89995cd3","Type":"ContainerStarted","Data":"80a87d733e68684cf41a62834cc2fe1e71599939cdad84e06433529d8387dd99"} Sep 16 13:54:04.146396 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:04.146356 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:54:04.459525 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:04.459481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" event={"ID":"f8d07bb8-f953-4e1d-abce-ee0c89995cd3","Type":"ContainerStarted","Data":"33f4f9422f54c2b4bcd9512abcdf6410aea3d54cceb2f2328c2e67a68d2943af"} Sep 16 13:54:04.484545 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:04.484491 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" podStartSLOduration=1.174693684 podStartE2EDuration="2.484474957s" totalImagesPullingTime="1.309781273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:54:04.482679569 +0000 UTC m=+663.523273524" watchObservedRunningTime="2026-09-16 13:54:04.484474957 +0000 UTC m=+663.525068912" Sep 16 13:54:09.354380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.354348 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s"] Sep 16 13:54:09.357785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.357765 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.362075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.362054 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 16 13:54:09.377615 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.377585 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s"] Sep 16 13:54:09.464354 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.464294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:09.466749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgbs\" (UniqueName: \"kubernetes.io/projected/e0272558-0ca2-4956-bab2-962b90a5818a-kube-api-access-djgbs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.466849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.466849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.466849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.466974 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0272558-0ca2-4956-bab2-962b90a5818a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.466974 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.466901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567539 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0272558-0ca2-4956-bab2-962b90a5818a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567944 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgbs\" (UniqueName: \"kubernetes.io/projected/e0272558-0ca2-4956-bab2-962b90a5818a-kube-api-access-djgbs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.567944 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.567778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.568055 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.568016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.568122 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.568076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.568350 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.568325 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.570104 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.570085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0272558-0ca2-4956-bab2-962b90a5818a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.570252 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.570236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0272558-0ca2-4956-bab2-962b90a5818a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.578420 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.578399 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgbs\" (UniqueName: \"kubernetes.io/projected/e0272558-0ca2-4956-bab2-962b90a5818a-kube-api-access-djgbs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s\" (UID: \"e0272558-0ca2-4956-bab2-962b90a5818a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.666674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.666597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:09.945918 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:09.945889 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s"] Sep 16 13:54:09.949519 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:54:09.949485 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0272558_0ca2_4956_bab2_962b90a5818a.slice/crio-53a49d4ce38ba69da3ee2932722cf55ba36e19b27bb0c5707d75e9e362089c80 WatchSource:0}: Error finding container 53a49d4ce38ba69da3ee2932722cf55ba36e19b27bb0c5707d75e9e362089c80: Status 404 returned error can't find the container with id 53a49d4ce38ba69da3ee2932722cf55ba36e19b27bb0c5707d75e9e362089c80 Sep 16 13:54:10.486911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:10.486804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" event={"ID":"e0272558-0ca2-4956-bab2-962b90a5818a","Type":"ContainerStarted","Data":"45ca0667f7f42856d5a85d9c6d08c5842562f54a1ec34ea5fe07459a9eea9cd0"} Sep 16 13:54:10.486911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:10.486850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" event={"ID":"e0272558-0ca2-4956-bab2-962b90a5818a","Type":"ContainerStarted","Data":"53a49d4ce38ba69da3ee2932722cf55ba36e19b27bb0c5707d75e9e362089c80"} Sep 16 13:54:10.510703 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:10.510646 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" podStartSLOduration=1.317244895 podStartE2EDuration="1.510627493s" totalImagesPullingTime="193.382598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:54:10.510288411 +0000 UTC m=+669.550882363" watchObservedRunningTime="2026-09-16 13:54:10.510627493 +0000 UTC m=+669.551221447" Sep 16 13:54:12.149905 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:12.149855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:54:12.478992 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:12.478957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:12.479204 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:12.479019 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:12.492089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:12.492061 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:12.508445 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:12.508413 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbdpg" Sep 16 13:54:19.667675 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:19.667629 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:19.668207 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:19.667694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:19.680133 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:19.680101 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:20.547063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:20.547030 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s" Sep 16 13:54:34.845860 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:34.845825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:34.961041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:34.961009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:56.592153 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:54:56.592066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:23.511129 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.511090 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54f85fdc87-ccwqh"] Sep 16 13:55:23.513509 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.513487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.542294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.542268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54f85fdc87-ccwqh"] Sep 16 13:55:23.589800 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.589767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44ea1263-b681-4bfe-ae12-73e036367a72-tls-cert\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.590085 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.590057 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxjr\" (UniqueName: \"kubernetes.io/projected/44ea1263-b681-4bfe-ae12-73e036367a72-kube-api-access-fmxjr\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.691392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.691362 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44ea1263-b681-4bfe-ae12-73e036367a72-tls-cert\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.691532 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.691438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxjr\" (UniqueName: \"kubernetes.io/projected/44ea1263-b681-4bfe-ae12-73e036367a72-kube-api-access-fmxjr\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.693653 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.693632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44ea1263-b681-4bfe-ae12-73e036367a72-tls-cert\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.706791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.706767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxjr\" (UniqueName: \"kubernetes.io/projected/44ea1263-b681-4bfe-ae12-73e036367a72-kube-api-access-fmxjr\") pod \"authorino-54f85fdc87-ccwqh\" (UID: \"44ea1263-b681-4bfe-ae12-73e036367a72\") " pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:23.822363 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:23.822287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54f85fdc87-ccwqh" Sep 16 13:55:24.038795 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:24.038762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:24.126653 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:24.126578 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54f85fdc87-ccwqh"] Sep 16 13:55:24.129427 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:24.129404 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ea1263_b681_4bfe_ae12_73e036367a72.slice/crio-edca07855a19cd71e9a5e730dc8880d08766a9fec61ddc5f5a0bf48ffd82b842 WatchSource:0}: Error finding container edca07855a19cd71e9a5e730dc8880d08766a9fec61ddc5f5a0bf48ffd82b842: Status 404 returned error can't find the container with id edca07855a19cd71e9a5e730dc8880d08766a9fec61ddc5f5a0bf48ffd82b842 Sep 16 13:55:24.815933 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:24.815894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54f85fdc87-ccwqh" event={"ID":"44ea1263-b681-4bfe-ae12-73e036367a72","Type":"ContainerStarted","Data":"edca07855a19cd71e9a5e730dc8880d08766a9fec61ddc5f5a0bf48ffd82b842"} Sep 16 13:55:25.293216 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.293176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:25.685392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.685357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:25.822368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.822328 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54f85fdc87-ccwqh" event={"ID":"44ea1263-b681-4bfe-ae12-73e036367a72","Type":"ContainerStarted","Data":"f5fa45e77005695d07d145f086ed05105e46066f3e4ca49cac7adebd48ce4128"} Sep 16 13:55:25.917762 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.917708 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54f85fdc87-ccwqh" podStartSLOduration=2.035039905 podStartE2EDuration="2.917692338s" totalImagesPullingTime="882.652433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:25.916619212 +0000 UTC m=+744.957213166" watchObservedRunningTime="2026-09-16 13:55:25.917692338 +0000 UTC m=+744.958286291" Sep 16 13:55:25.991234 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.991197 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:55:25.991463 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:25.991434 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" podUID="9fb934be-df2d-46c6-a73b-4d9e936282ee" containerName="authorino" containerID="cri-o://f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8" gracePeriod=30 Sep 16 13:55:26.239595 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.239567 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:55:26.318937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.318844 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert\") pod \"9fb934be-df2d-46c6-a73b-4d9e936282ee\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " Sep 16 13:55:26.318937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.318934 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rc2l\" (UniqueName: \"kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l\") pod \"9fb934be-df2d-46c6-a73b-4d9e936282ee\" (UID: \"9fb934be-df2d-46c6-a73b-4d9e936282ee\") " Sep 16 13:55:26.320980 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.320943 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l" (OuterVolumeSpecName: "kube-api-access-9rc2l") pod "9fb934be-df2d-46c6-a73b-4d9e936282ee" (UID: "9fb934be-df2d-46c6-a73b-4d9e936282ee"). InnerVolumeSpecName "kube-api-access-9rc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:55:26.330179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.330148 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9fb934be-df2d-46c6-a73b-4d9e936282ee" (UID: "9fb934be-df2d-46c6-a73b-4d9e936282ee"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:55:26.419929 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.419859 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fb934be-df2d-46c6-a73b-4d9e936282ee-tls-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:55:26.419929 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.419926 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rc2l\" (UniqueName: \"kubernetes.io/projected/9fb934be-df2d-46c6-a73b-4d9e936282ee-kube-api-access-9rc2l\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:55:26.829967 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.829931 2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb934be-df2d-46c6-a73b-4d9e936282ee" containerID="f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8" exitCode=0 Sep 16 13:55:26.830415 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.829979 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" Sep 16 13:55:26.830415 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.830031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" event={"ID":"9fb934be-df2d-46c6-a73b-4d9e936282ee","Type":"ContainerDied","Data":"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8"} Sep 16 13:55:26.830415 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.830083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d4d9f959d-r4sdd" event={"ID":"9fb934be-df2d-46c6-a73b-4d9e936282ee","Type":"ContainerDied","Data":"90ac1c80915a22cd7f4f45c65aa7c913b60c157ad92a87d9d3bad16b20b97e15"} Sep 16 13:55:26.830415 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.830106 2559 scope.go:117] "RemoveContainer" containerID="f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8" Sep 16 13:55:26.841616 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.841588 2559 scope.go:117] "RemoveContainer" containerID="f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8" Sep 16 13:55:26.841958 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:26.841938 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8\": container with ID starting with f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8 not found: ID does not exist" containerID="f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8" Sep 16 13:55:26.842027 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.841967 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8"} err="failed to get container status \"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8\": rpc error: code = NotFound desc = could not find container \"f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8\": container with ID starting with f08a2c6a92fe0f87d069d95ebb35e3d80124b2e27eca8aac81a6490d07ba52c8 not found: ID does not exist" Sep 16 13:55:26.865957 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.865924 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:55:26.885734 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:26.885704 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d4d9f959d-r4sdd"] Sep 16 13:55:27.472024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:27.471991 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb934be-df2d-46c6-a73b-4d9e936282ee" path="/var/lib/kubelet/pods/9fb934be-df2d-46c6-a73b-4d9e936282ee/volumes" Sep 16 13:55:42.843662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:42.843624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:42.977575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:42.977542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:43.150335 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:43.150233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:43.341036 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:43.341005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:43.470171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:43.470144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:43.574738 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:43.574703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:43.669506 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:43.669476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:44.242855 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.242814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:44.342939 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.342908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:44.731847 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.731814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:44.879831 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.879800 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7908-e6c68b"] Sep 16 13:55:44.880157 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.880146 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb934be-df2d-46c6-a73b-4d9e936282ee" containerName="authorino" Sep 16 13:55:44.880202 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.880159 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb934be-df2d-46c6-a73b-4d9e936282ee" containerName="authorino" Sep 16 13:55:44.880235 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.880214 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb934be-df2d-46c6-a73b-4d9e936282ee" containerName="authorino" Sep 16 13:55:44.882742 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.882725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:44.886492 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.886472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 13:55:44.903108 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.903079 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7908-e6c68b"] Sep 16 13:55:44.989095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:44.989018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8f4\" (UniqueName: \"kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4\") pod \"test-curl-7908-e6c68b\" (UID: \"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e\") " pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:45.090314 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:45.090280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8f4\" (UniqueName: \"kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4\") pod \"test-curl-7908-e6c68b\" (UID: \"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e\") " pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:45.100806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:45.100780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8f4\" (UniqueName: \"kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4\") pod \"test-curl-7908-e6c68b\" (UID: \"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e\") " pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:45.190922 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:45.190850 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:45.467976 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:45.467950 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4b8de3_1577_43e4_ab67_cfcd8fefdd9e.slice/crio-7e0a53f1c12200918d3a98d2d0d12ef1f3bb92e4521fa90d335b0e30fdae6472 WatchSource:0}: Error finding container 7e0a53f1c12200918d3a98d2d0d12ef1f3bb92e4521fa90d335b0e30fdae6472: Status 404 returned error can't find the container with id 7e0a53f1c12200918d3a98d2d0d12ef1f3bb92e4521fa90d335b0e30fdae6472 Sep 16 13:55:45.472455 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:45.472431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7908-e6c68b"] Sep 16 13:55:45.917786 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:45.917750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7908-e6c68b" event={"ID":"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e","Type":"ContainerStarted","Data":"7e0a53f1c12200918d3a98d2d0d12ef1f3bb92e4521fa90d335b0e30fdae6472"} Sep 16 13:55:46.497713 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.497607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:46.604425 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.604389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:46.701154 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.701124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:46.764367 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.764274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:55:46.924525 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.924485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7908-e6c68b" event={"ID":"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e","Type":"ContainerStarted","Data":"79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052"} Sep 16 13:55:46.949447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:46.949396 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7908-e6c68b" podStartSLOduration=2.219019558 podStartE2EDuration="2.949378603s" totalImagesPullingTime="730.359045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:46.947615938 +0000 UTC m=+765.988209913" watchObservedRunningTime="2026-09-16 13:55:46.949378603 +0000 UTC m=+765.989972589" Sep 16 13:55:47.653241 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.653208 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7908-e6c68b"] Sep 16 13:55:47.661476 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.661447 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7908-e6c68b"] Sep 16 13:55:47.742766 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.742731 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:47.794611 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.794582 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7908-2fed9c"] Sep 16 13:55:47.794976 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.794961 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" containerName="test-curl-7908-e6c68b" Sep 16 13:55:47.795025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.794977 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" containerName="test-curl-7908-e6c68b" Sep 16 13:55:47.795059 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.795034 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" containerName="test-curl-7908-e6c68b" Sep 16 13:55:47.797004 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.796988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:47.815661 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.815637 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7908-2fed9c"] Sep 16 13:55:47.891998 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.891962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:47.915915 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:47.915888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7rr\" (UniqueName: \"kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr\") pod \"test-curl-7908-2fed9c\" (UID: \"8f3e9206-d35b-4da2-a352-d7c6913345ca\") " pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:47.928762 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:47.928733 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7908-e6c68b" container="test-curl-7908-e6c68b" Sep 16 13:55:48.016917 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.016812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7rr\" (UniqueName: \"kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr\") pod \"test-curl-7908-2fed9c\" (UID: \"8f3e9206-d35b-4da2-a352-d7c6913345ca\") " pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:48.047072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.047042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7rr\" (UniqueName: \"kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr\") pod \"test-curl-7908-2fed9c\" (UID: \"8f3e9206-d35b-4da2-a352-d7c6913345ca\") " pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:48.085130 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.085102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:48.105051 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.105010 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:48.374576 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.374488 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7908-2fed9c"] Sep 16 13:55:48.378069 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:48.378033 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f3e9206_d35b_4da2_a352_d7c6913345ca.slice/crio-65d6d445d1d14f66c260eeba3739065a040b88e44b7c610cc603b79e2b7a5049 WatchSource:0}: Error finding container 65d6d445d1d14f66c260eeba3739065a040b88e44b7c610cc603b79e2b7a5049: Status 404 returned error can't find the container with id 65d6d445d1d14f66c260eeba3739065a040b88e44b7c610cc603b79e2b7a5049 Sep 16 13:55:48.559223 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.559190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:48.661586 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.661467 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:48.934431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.934338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7908-2fed9c" event={"ID":"8f3e9206-d35b-4da2-a352-d7c6913345ca","Type":"ContainerStarted","Data":"e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e"} Sep 16 13:55:48.934431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.934389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7908-2fed9c" event={"ID":"8f3e9206-d35b-4da2-a352-d7c6913345ca","Type":"ContainerStarted","Data":"65d6d445d1d14f66c260eeba3739065a040b88e44b7c610cc603b79e2b7a5049"} Sep 16 13:55:48.934611 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.934488 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7908-e6c68b" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" containerName="test-curl-7908-e6c68b" containerID="cri-o://79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052" gracePeriod=2 Sep 16 13:55:48.955861 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:48.955809 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7908-2fed9c" podStartSLOduration=1.95579379 podStartE2EDuration="1.95579379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:48.954387737 +0000 UTC m=+767.994981691" watchObservedRunningTime="2026-09-16 13:55:48.95579379 +0000 UTC m=+767.996387744" Sep 16 13:55:48.956268 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:48.956235 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-e6c68b\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:49.275324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.275287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.376436 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.376403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.645537 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.645452 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7908-2fed9c"] Sep 16 13:55:49.656964 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.656927 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7908-2fed9c"] Sep 16 13:55:49.759839 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.759803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.914675 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:49.914581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.939553 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:49.939525 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7908-2fed9c" container="test-curl-7908-2fed9c" Sep 16 13:55:50.109607 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.109567 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.240103 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.240072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.305886 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.305828 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 13:55:50.306218 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.306203 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" containerName="test-curl-7908-2fed9c" Sep 16 13:55:50.306267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.306219 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" containerName="test-curl-7908-2fed9c" Sep 16 13:55:50.306300 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.306294 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" containerName="test-curl-7908-2fed9c" Sep 16 13:55:50.308274 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.308257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.311739 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.311706 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b52a89-dockercfg-mm7j2\"" Sep 16 13:55:50.312155 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.312137 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b52a89\"" Sep 16 13:55:50.319287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.319266 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 13:55:50.338934 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.338907 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.339045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.338976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmmc\" (UniqueName: \"kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.440159 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.440119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmmc\" (UniqueName: \"kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.440306 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.440182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.440806 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.440787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.450350 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.450323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmmc\" (UniqueName: \"kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc\") pod \"payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.560583 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.560506 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:55:50.562825 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.562808 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.568737 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.568707 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:55:50.568887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.568770 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-617597-openshift-default-dockercfg-s9czr\"" Sep 16 13:55:50.602441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.602397 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:55:50.617999 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.617962 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:50.639546 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.639504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.642552 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642552 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f67c\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642965 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.642965 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.642926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f67c\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744407 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744526 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744898 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.744898 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.744806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.745052 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.745021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.745102 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.745057 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.745136 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.745116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.745255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.745237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.746687 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.746661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.746962 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.746943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.754963 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.754938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f67c\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.760575 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.760557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token\") pod \"e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.866035 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.865955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.873934 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.873907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:50.892673 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.892651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 13:55:50.894818 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:50.894794 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b11736_4bf1_4536_9914_1434c18c7533.slice/crio-b744e3cd4b6b94179cfb12b4d7de705c389a06ae4a83f1ddd3dc0c61ebb8bba8 WatchSource:0}: Error finding container b744e3cd4b6b94179cfb12b4d7de705c389a06ae4a83f1ddd3dc0c61ebb8bba8: Status 404 returned error can't find the container with id b744e3cd4b6b94179cfb12b4d7de705c389a06ae4a83f1ddd3dc0c61ebb8bba8 Sep 16 13:55:50.947534 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.947504 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" event={"ID":"12b11736-4bf1-4536-9914-1434c18c7533","Type":"ContainerStarted","Data":"b744e3cd4b6b94179cfb12b4d7de705c389a06ae4a83f1ddd3dc0c61ebb8bba8"} Sep 16 13:55:50.947635 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.947564 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7908-2fed9c" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" containerName="test-curl-7908-2fed9c" containerID="cri-o://e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e" gracePeriod=2 Sep 16 13:55:50.949896 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:50.949857 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:50.963767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:50.963743 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:51.117807 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.117749 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:51.120012 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:51.119984 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:51.147952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.147925 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl8f4\" (UniqueName: \"kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4\") pod \"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e\" (UID: \"7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e\") " Sep 16 13:55:51.149330 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.149305 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:55:51.150063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.150035 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4" (OuterVolumeSpecName: "kube-api-access-vl8f4") pod "7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" (UID: "7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e"). InnerVolumeSpecName "kube-api-access-vl8f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:55:51.151338 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:51.151314 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51aa693_e4d3_40ad_80e0_bb92455fa620.slice/crio-c6d0778a9cea991e798f6333b81b373fd12958e952e84c7de7ae002ec0e5e398 WatchSource:0}: Error finding container c6d0778a9cea991e798f6333b81b373fd12958e952e84c7de7ae002ec0e5e398: Status 404 returned error can't find the container with id c6d0778a9cea991e798f6333b81b373fd12958e952e84c7de7ae002ec0e5e398 Sep 16 13:55:51.248767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.248726 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8f4\" (UniqueName: \"kubernetes.io/projected/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e-kube-api-access-vl8f4\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:55:51.469178 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:51.469141 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:51.471842 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.471822 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" path="/var/lib/kubelet/pods/7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e/volumes" Sep 16 13:55:51.931623 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.931534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:51.964133 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.964090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerStarted","Data":"c6d0778a9cea991e798f6333b81b373fd12958e952e84c7de7ae002ec0e5e398"} Sep 16 13:55:51.965799 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.965769 2559 generic.go:356] "Generic (PLEG): container finished" podID="7e4b8de3-1577-43e4-ab67-cfcd8fefdd9e" containerID="79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052" exitCode=137 Sep 16 13:55:51.965989 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.965918 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-e6c68b" Sep 16 13:55:51.966096 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:51.965947 2559 scope.go:117] "RemoveContainer" containerID="79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052" Sep 16 13:55:52.290261 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:52.289993 2559 scope.go:117] "RemoveContainer" containerID="79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052" Sep 16 13:55:52.290452 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:52.290428 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052\": container with ID starting with 79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052 not found: ID does not exist" containerID="79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052" Sep 16 13:55:52.290530 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:52.290465 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052"} err="failed to get container status \"79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052\": rpc error: code = NotFound desc = could not find container \"79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052\": container with ID starting with 79d396f46d5cc1086bad829eaa473efdb2880a87c117ffcd367fbec146851052 not found: ID does not exist" Sep 16 13:55:52.949811 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:52.949770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:52.974908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:52.974626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" event={"ID":"12b11736-4bf1-4536-9914-1434c18c7533","Type":"ContainerStarted","Data":"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638"} Sep 16 13:55:52.974908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:52.974830 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:55:53.009332 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.009271 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" podStartSLOduration=1.590289941 podStartE2EDuration="3.00924862s" totalImagesPullingTime="1.418958679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:53.009011555 +0000 UTC m=+772.049605511" watchObservedRunningTime="2026-09-16 13:55:53.00924862 +0000 UTC m=+772.049842575" Sep 16 13:55:53.647130 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.647080 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:53.647258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.647180 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:53.647258 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.647227 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:53.733296 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.733271 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:53.772665 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.772630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q7rr\" (UniqueName: \"kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr\") pod \"8f3e9206-d35b-4da2-a352-d7c6913345ca\" (UID: \"8f3e9206-d35b-4da2-a352-d7c6913345ca\") " Sep 16 13:55:53.774619 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.774585 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr" (OuterVolumeSpecName: "kube-api-access-8q7rr") pod "8f3e9206-d35b-4da2-a352-d7c6913345ca" (UID: "8f3e9206-d35b-4da2-a352-d7c6913345ca"). InnerVolumeSpecName "kube-api-access-8q7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:55:53.874071 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.873990 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q7rr\" (UniqueName: \"kubernetes.io/projected/8f3e9206-d35b-4da2-a352-d7c6913345ca-kube-api-access-8q7rr\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:55:53.919307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.919268 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:53.982685 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.982645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerStarted","Data":"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9"} Sep 16 13:55:53.984166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.984142 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f3e9206-d35b-4da2-a352-d7c6913345ca" containerID="e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e" exitCode=137 Sep 16 13:55:53.984301 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.984187 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:53.984301 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.984244 2559 scope.go:117] "RemoveContainer" containerID="e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e" Sep 16 13:55:53.993361 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.993341 2559 scope.go:117] "RemoveContainer" containerID="e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e" Sep 16 13:55:53.993648 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:53.993624 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e\": container with ID starting with e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e not found: ID does not exist" containerID="e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e" Sep 16 13:55:53.993723 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:53.993662 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e"} err="failed to get container status \"e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e\": rpc error: code = NotFound desc = could not find container \"e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e\": container with ID starting with e5f20ac539ec83574ffc1e8210cf1e2f3454d1b350daab4737155ec3eff9be3e not found: ID does not exist" Sep 16 13:55:54.013819 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.013763 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podStartSLOduration=1.520582116 podStartE2EDuration="4.013741196s" totalImagesPullingTime="2.49315908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:54.013123131 +0000 UTC m=+773.053717089" watchObservedRunningTime="2026-09-16 13:55:54.013741196 +0000 UTC m=+773.054335151" Sep 16 13:55:54.015212 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:54.015182 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:54.017037 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.017009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.372802 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.372766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.524517 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.524477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.669503 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.669414 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.874280 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.874247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:54.875455 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.875427 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.879780 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.879754 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:54.881914 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:54.881890 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:54.966236 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.966205 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 13:55:54.969053 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.969037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:54.975779 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.975751 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-035c64\"" Sep 16 13:55:54.976723 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.976701 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-035c64-dockercfg-fhccl\"" Sep 16 13:55:54.977062 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.977045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.989947 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.989920 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:54.990807 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.990753 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:55:54.995410 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:54.995386 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 13:55:55.020345 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:55.020310 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:55.023066 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:55.023039 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:55.085969 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.085931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrwf\" (UniqueName: \"kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.086208 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.086053 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.130184 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.130152 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:55:55.132548 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.132527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.134457 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:55:55.134427 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7908-2fed9c\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" pod="opendatahub/test-curl-7908-2fed9c" Sep 16 13:55:55.134781 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.134764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1dcc64e3-openshift-default-dockercfg-q8n5t\"" Sep 16 13:55:55.153490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.153459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:55:55.186723 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.186887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.186887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.186887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrwf\" (UniqueName: \"kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.186887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.186887 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.186992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbhr\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.187016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.187048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.187124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.187165 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.187146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.187944 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.187909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.199581 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.199554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrwf\" (UniqueName: \"kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf\") pod \"payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.280269 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.280170 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:55.288467 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288832 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288832 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288832 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.288832 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbhr\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.289083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.288883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.289202 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.289173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.289271 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.289216 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.289330 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.289262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.289482 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.289463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.291169 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.291148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.291394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.291376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.298502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.298475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.298645 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.298622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbhr\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr\") pod \"e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.442444 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.442411 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:55.471830 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.471791 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3e9206-d35b-4da2-a352-d7c6913345ca" path="/var/lib/kubelet/pods/8f3e9206-d35b-4da2-a352-d7c6913345ca/volumes" Sep 16 13:55:55.552813 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.552787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 13:55:55.554502 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:55.554475 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d306d8c_0c25_4f19_9b8c_e425d3fd4634.slice/crio-d125c0ddb6381b64b62d5ac68eb5772891f92017a924b68ae96fc91b4dee2b45 WatchSource:0}: Error finding container d125c0ddb6381b64b62d5ac68eb5772891f92017a924b68ae96fc91b4dee2b45: Status 404 returned error can't find the container with id d125c0ddb6381b64b62d5ac68eb5772891f92017a924b68ae96fc91b4dee2b45 Sep 16 13:55:55.605845 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.605557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:55.714460 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.714435 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:55:55.717123 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:55:55.717091 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabf4894_4622_4581_8525_7da70abcfa06.slice/crio-8495c48625ffed886e7aa1765e1b3e14379574c9c057cacf59f7314a87237afc WatchSource:0}: Error finding container 8495c48625ffed886e7aa1765e1b3e14379574c9c057cacf59f7314a87237afc: Status 404 returned error can't find the container with id 8495c48625ffed886e7aa1765e1b3e14379574c9c057cacf59f7314a87237afc Sep 16 13:55:55.720285 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.720256 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:55.720347 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.720316 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:55.720389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.720347 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:55:55.871809 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.871714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:55.996603 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.996559 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerStarted","Data":"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b"} Sep 16 13:55:55.997100 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.996622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerStarted","Data":"8495c48625ffed886e7aa1765e1b3e14379574c9c057cacf59f7314a87237afc"} Sep 16 13:55:55.998971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.998572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" event={"ID":"1d306d8c-0c25-4f19-9b8c-e425d3fd4634","Type":"ContainerStarted","Data":"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122"} Sep 16 13:55:55.998971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.998622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" event={"ID":"1d306d8c-0c25-4f19-9b8c-e425d3fd4634","Type":"ContainerStarted","Data":"d125c0ddb6381b64b62d5ac68eb5772891f92017a924b68ae96fc91b4dee2b45"} Sep 16 13:55:55.998971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:55.998746 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:55:56.018545 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.018493 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podStartSLOduration=1.018477361 podStartE2EDuration="1.018477361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:56.018062638 +0000 UTC m=+775.058656592" watchObservedRunningTime="2026-09-16 13:55:56.018477361 +0000 UTC m=+775.059071314" Sep 16 13:55:56.051343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.051275 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" podStartSLOduration=2.051252604 podStartE2EDuration="2.051252604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:56.049646303 +0000 UTC m=+775.090240257" watchObservedRunningTime="2026-09-16 13:55:56.051252604 +0000 UTC m=+775.091846558" Sep 16 13:55:56.065170 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.065134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:56.164235 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.164145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:56.443451 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.443424 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:56.448558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.448529 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:56.865123 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:56.865044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:57.003746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:57.003699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:57.004771 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:57.004748 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:55:57.542946 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:57.542912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:57.809427 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:57.809331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:58.235128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:58.235091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:59.381472 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:55:59.381427 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.304931 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:00.304891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.387339 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:00.387291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.488124 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:00.488093 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.853785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:00.853752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.948456 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:00.948420 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.053933 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.053896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:56:01.057318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.057301 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.059712 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.059690 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-617597-dockercfg-tpn8n\"" Sep 16 13:56:01.059830 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.059692 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-617597\"" Sep 16 13:56:01.067453 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.067429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:56:01.075451 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.075426 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.140720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.140631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd4j\" (UniqueName: \"kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.140720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.140682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.149146 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.149114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.241818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.241778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd4j\" (UniqueName: \"kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.242022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.241830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.242516 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.242497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.253115 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.253088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd4j\" (UniqueName: \"kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j\") pod \"payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.371041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.371001 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-617597-dockercfg-tpn8n\"" Sep 16 13:56:01.379063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.379033 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:01.422099 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.422011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.524345 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.524311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.622993 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.622956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.671963 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.671937 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:56:01.672551 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:01.672496 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7d9db1_49b2_4f29_b3f9_751da67cf8cc.slice/crio-25409fae97a4c0503401455cf5af7969c3b60941ed7135c629e3101795975304 WatchSource:0}: Error finding container 25409fae97a4c0503401455cf5af7969c3b60941ed7135c629e3101795975304: Status 404 returned error can't find the container with id 25409fae97a4c0503401455cf5af7969c3b60941ed7135c629e3101795975304 Sep 16 13:56:01.858276 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:01.858239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.023189 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.023152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.028707 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.028680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" event={"ID":"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc","Type":"ContainerStarted","Data":"5e6b72532a8d2db62da83f5f6a4654cea6ea849a8422d9d4b40a3ce493002eab"} Sep 16 13:56:02.028827 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.028735 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:02.028827 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.028758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" event={"ID":"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc","Type":"ContainerStarted","Data":"25409fae97a4c0503401455cf5af7969c3b60941ed7135c629e3101795975304"} Sep 16 13:56:02.050733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.050680 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" podStartSLOduration=1.050666813 podStartE2EDuration="1.050666813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:02.048796991 +0000 UTC m=+781.089390946" watchObservedRunningTime="2026-09-16 13:56:02.050666813 +0000 UTC m=+781.091260767" Sep 16 13:56:02.220863 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.220829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.323542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.323447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.424272 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.424235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.469507 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.469478 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:02.472984 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.472967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.477952 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.477933 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1dcc64e3\"" Sep 16 13:56:02.478058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.478030 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1dcc64e3-dockercfg-psqgw\"" Sep 16 13:56:02.490818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.490790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:02.553113 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.553082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.553272 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.553140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqkp\" (UniqueName: \"kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.654561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.654475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.654561 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.654529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqkp\" (UniqueName: \"kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.655174 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.655155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.664048 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.664024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqkp\" (UniqueName: \"kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp\") pod \"payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.783763 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.783728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:02.823203 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:02.823170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.076948 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.076921 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:03.078100 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:03.078063 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ab6a10_ee9d_45d9_9df7_5e92b2a91296.slice/crio-cc06417a44026dc2ffcf9a82d63964ee23964a3de92231e673ae6b853822fe03 WatchSource:0}: Error finding container cc06417a44026dc2ffcf9a82d63964ee23964a3de92231e673ae6b853822fe03: Status 404 returned error can't find the container with id cc06417a44026dc2ffcf9a82d63964ee23964a3de92231e673ae6b853822fe03 Sep 16 13:56:03.373404 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.373318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.516358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.516318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.636415 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.636336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.733808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.733780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.986722 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:03.986688 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 13:56:04.039634 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.039600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" event={"ID":"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296","Type":"ContainerStarted","Data":"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91"} Sep 16 13:56:04.039813 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.039651 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" event={"ID":"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296","Type":"ContainerStarted","Data":"cc06417a44026dc2ffcf9a82d63964ee23964a3de92231e673ae6b853822fe03"} Sep 16 13:56:04.039813 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.039747 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:04.061042 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.060990 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" podStartSLOduration=2.060972894 podStartE2EDuration="2.060972894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:04.060572883 +0000 UTC m=+783.101166838" watchObservedRunningTime="2026-09-16 13:56:04.060972894 +0000 UTC m=+783.101566847" Sep 16 13:56:04.132569 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.132529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:04.988199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:04.988164 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:56:06.090465 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:06.090431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:06.287921 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:06.287862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:06.510940 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:06.510859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:06.643646 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:06.643607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:07.005673 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:07.005645 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 13:56:08.093499 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:08.093459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:09.366096 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:09.366037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:10.677566 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:10.677516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:11.207034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:11.206996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:11.800792 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:11.800691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:11.898431 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:11.898398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.279399 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.279366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.594517 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.594436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.698910 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.698861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.743996 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.743969 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9"] Sep 16 13:56:12.755912 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.755861 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:12.759199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.759173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:56:12.760475 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.760441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9"] Sep 16 13:56:12.766138 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.765948 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 13:56:12.810748 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.810707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.862467 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.862366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9\" (UID: \"143e9d4f-0d60-4766-9edb-9c2dfb4b8396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:12.905251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.905211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:12.963440 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.963393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9\" (UID: \"143e9d4f-0d60-4766-9edb-9c2dfb4b8396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:12.964337 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.964305 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9\" (UID: \"143e9d4f-0d60-4766-9edb-9c2dfb4b8396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:12.967060 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.967033 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 13:56:12.975107 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.975081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:12.979574 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.979543 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-95743e-dockercfg-q5s8p\"" Sep 16 13:56:12.979763 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.979741 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-95743e\"" Sep 16 13:56:12.986146 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:12.986121 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 13:56:13.035590 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.035563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:56:13.036676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.036652 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 13:56:13.040721 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.040705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.067279 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.064458 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 13:56:13.075903 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.072428 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.075903 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.072485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6tn\" (UniqueName: \"kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.086388 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.086352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:13.180409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.178434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.180409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.178483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2f6\" (UniqueName: \"kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.180409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.178545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6tn\" (UniqueName: \"kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.180409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.178611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.180409 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.179830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.191818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.191748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6tn\" (UniqueName: \"kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn\") pod \"payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.279710 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.279664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.279933 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.279779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2f6\" (UniqueName: \"kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.280554 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.280525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.286588 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.286559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:13.289812 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.289787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2f6\" (UniqueName: \"kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6\") pod \"payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.356419 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.356350 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:13.384094 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.382092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:13.392578 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.391918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9"] Sep 16 13:56:13.398908 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:13.398850 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143e9d4f_0d60_4766_9edb_9c2dfb4b8396.slice/crio-4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63 WatchSource:0}: Error finding container 4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63: Status 404 returned error can't find the container with id 4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63 Sep 16 13:56:13.577164 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.577059 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 13:56:13.586518 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:13.586486 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0d306e_7153_4867_bf21_44c0533553d5.slice/crio-43d30c342cdcd7ef4665a60f098c8a6bfbcc86f9de4c65f99f86c1c342f9cc80 WatchSource:0}: Error finding container 43d30c342cdcd7ef4665a60f098c8a6bfbcc86f9de4c65f99f86c1c342f9cc80: Status 404 returned error can't find the container with id 43d30c342cdcd7ef4665a60f098c8a6bfbcc86f9de4c65f99f86c1c342f9cc80 Sep 16 13:56:13.668411 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:13.668243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 13:56:13.670791 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:13.670758 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925f7d04_83f3_4b19_a3f2_ef60c3af5de1.slice/crio-7bac3221df427388adfc9829a43d49141be4f8633be8dc363475c21445090e70 WatchSource:0}: Error finding container 7bac3221df427388adfc9829a43d49141be4f8633be8dc363475c21445090e70: Status 404 returned error can't find the container with id 7bac3221df427388adfc9829a43d49141be4f8633be8dc363475c21445090e70 Sep 16 13:56:14.113176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.113137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" event={"ID":"925f7d04-83f3-4b19-a3f2-ef60c3af5de1","Type":"ContainerStarted","Data":"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4"} Sep 16 13:56:14.113625 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.113196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" event={"ID":"925f7d04-83f3-4b19-a3f2-ef60c3af5de1","Type":"ContainerStarted","Data":"7bac3221df427388adfc9829a43d49141be4f8633be8dc363475c21445090e70"} Sep 16 13:56:14.113625 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.113250 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:14.115379 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.115268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" event={"ID":"dd0d306e-7153-4867-bf21-44c0533553d5","Type":"ContainerStarted","Data":"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769"} Sep 16 13:56:14.115379 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.115321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" event={"ID":"dd0d306e-7153-4867-bf21-44c0533553d5","Type":"ContainerStarted","Data":"43d30c342cdcd7ef4665a60f098c8a6bfbcc86f9de4c65f99f86c1c342f9cc80"} Sep 16 13:56:14.115379 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.115348 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:14.117487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.117458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" event={"ID":"143e9d4f-0d60-4766-9edb-9c2dfb4b8396","Type":"ContainerStarted","Data":"4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63"} Sep 16 13:56:14.136006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.135352 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" podStartSLOduration=1.135330882 podStartE2EDuration="1.135330882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:14.134467926 +0000 UTC m=+793.175061881" watchObservedRunningTime="2026-09-16 13:56:14.135330882 +0000 UTC m=+793.175924834" Sep 16 13:56:14.162355 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.162289 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" podStartSLOduration=2.162268579 podStartE2EDuration="2.162268579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:14.157391714 +0000 UTC m=+793.197985692" watchObservedRunningTime="2026-09-16 13:56:14.162268579 +0000 UTC m=+793.202862534" Sep 16 13:56:14.529987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.529923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.648466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:14.648432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.047218 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:15.047187 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:15.133485 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:15.133443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.271120 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:15.271085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.570116 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:15.570056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.808174 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:15.808142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:16.462671 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:16.462631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:17.138071 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.137972 2559 generic.go:356] "Generic (PLEG): container finished" podID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerID="4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9" exitCode=0 Sep 16 13:56:17.138071 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.138058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerDied","Data":"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9"} Sep 16 13:56:17.138580 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.138549 2559 scope.go:117] "RemoveContainer" containerID="4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9" Sep 16 13:56:17.139980 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.139936 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.140095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.140047 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.140095 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.140079 2559 generic.go:356] "Generic (PLEG): container finished" podID="eabf4894-4622-4581-8525-7da70abcfa06" containerID="aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b" exitCode=0 Sep 16 13:56:17.140179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.140096 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.140179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.140129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerDied","Data":"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b"} Sep 16 13:56:17.140606 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.140590 2559 scope.go:117] "RemoveContainer" containerID="aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b" Sep 16 13:56:17.141920 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.141892 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.142027 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.141950 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.142027 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.141987 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:17.621785 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:17.621744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:18.147094 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.147052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerStarted","Data":"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b"} Sep 16 13:56:18.149574 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.149540 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerStarted","Data":"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1"} Sep 16 13:56:18.443142 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.443004 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:18.443906 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.443845 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:18.444055 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.443940 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:18.551746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.551700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:18.874827 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.874773 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:56:18.875357 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.875195 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 16 13:56:18.875357 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:18.875247 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 16 13:56:19.443748 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:19.443700 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:19.443979 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:19.443766 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:19.641047 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:19.641011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:19.875251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:19.875144 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 16 13:56:19.875251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:19.875226 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 16 13:56:20.443591 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.443547 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:20.443803 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.443616 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:20.636570 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.636513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:20.747518 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.747425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:20.874740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.874695 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 16 13:56:20.874983 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.874776 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 16 13:56:20.878991 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.878953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:20.998690 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:20.998612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:21.443327 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:21.443290 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:21.443503 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:21.443357 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:21.810852 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:21.810769 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:21.874522 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:21.874483 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 16 13:56:21.874797 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:21.874767 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 16 13:56:22.443849 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:22.443805 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:22.444287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:22.443860 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:22.875080 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:22.874972 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Sep 16 13:56:22.875380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:22.875328 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Sep 16 13:56:22.917788 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:22.917757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.443256 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.443211 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 16 13:56:23.443472 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.443277 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 16 13:56:23.736758 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.736639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.878288 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.878252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:56:23.878735 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.878715 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:56:23.879471 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:23.879453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:56:24.448012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:24.447980 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:24.448392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:24.448366 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:24.448951 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:24.448933 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:24.541948 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:24.541907 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:25.126944 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:25.126910 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 13:56:25.127394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:25.126991 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 13:56:25.473972 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:25.473942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:25.575532 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:25.575499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:25.650090 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:25.650052 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:56:26.572181 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:26.572142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:27.655616 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:27.655584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:28.217823 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:28.217739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" event={"ID":"143e9d4f-0d60-4766-9edb-9c2dfb4b8396","Type":"ContainerStarted","Data":"47674f840f8abdc390f1cb7b503d1fdc0691132838793c47441f0e7d77c6a9ef"} Sep 16 13:56:28.249933 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:28.249857 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" podStartSLOduration=1.987007905 podStartE2EDuration="16.249839696s" totalImagesPullingTime="14.262831791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:28.248075122 +0000 UTC m=+807.288669076" watchObservedRunningTime="2026-09-16 13:56:28.249839696 +0000 UTC m=+807.290433652" Sep 16 13:56:28.250273 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:28.250253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:28.352151 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:28.352119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:28.717517 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:28.717481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:29.224514 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:29.224481 2559 generic.go:356] "Generic (PLEG): container finished" podID="143e9d4f-0d60-4766-9edb-9c2dfb4b8396" containerID="47674f840f8abdc390f1cb7b503d1fdc0691132838793c47441f0e7d77c6a9ef" exitCode=0 Sep 16 13:56:29.224709 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:29.224567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" event={"ID":"143e9d4f-0d60-4766-9edb-9c2dfb4b8396","Type":"ContainerDied","Data":"47674f840f8abdc390f1cb7b503d1fdc0691132838793c47441f0e7d77c6a9ef"} Sep 16 13:56:30.359394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:30.359368 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:30.475208 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:30.475172 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca\") pod \"143e9d4f-0d60-4766-9edb-9c2dfb4b8396\" (UID: \"143e9d4f-0d60-4766-9edb-9c2dfb4b8396\") " Sep 16 13:56:30.475557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:30.475532 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "143e9d4f-0d60-4766-9edb-9c2dfb4b8396" (UID: "143e9d4f-0d60-4766-9edb-9c2dfb4b8396"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:30.576150 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:30.576058 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/143e9d4f-0d60-4766-9edb-9c2dfb4b8396-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:31.235473 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:31.235441 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" Sep 16 13:56:31.235473 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:31.235455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9" event={"ID":"143e9d4f-0d60-4766-9edb-9c2dfb4b8396","Type":"ContainerDied","Data":"4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63"} Sep 16 13:56:31.235746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:31.235504 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed5cfa81c61ae21a1d35eb45d0dfb89fc3e56e5d6ee1901086d0584ba72ed63" Sep 16 13:56:32.986653 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:32.986602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:33.802976 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:33.802943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:34.535094 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.535064 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:34.582615 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.582579 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:34.583043 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.583008 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" podUID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" containerName="payload-pre-processing" containerID="cri-o://894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91" gracePeriod=30 Sep 16 13:56:34.636091 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.636051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:34.736099 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.736057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:34.770145 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.769888 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9"] Sep 16 13:56:34.773678 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.773627 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dcc64e3-6dqz9"] Sep 16 13:56:34.836081 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.836050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:34.847929 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.847902 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:34.909212 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.909179 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:56:34.909416 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.909396 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" containerID="cri-o://8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1" gracePeriod=30 Sep 16 13:56:34.923856 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.923817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtqkp\" (UniqueName: \"kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp\") pod \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " Sep 16 13:56:34.924042 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.923930 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume\") pod \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\" (UID: \"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296\") " Sep 16 13:56:34.924324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.924301 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" (UID: "d8ab6a10-ee9d-45d9-9df7-5e92b2a91296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:34.925980 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.925953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp" (OuterVolumeSpecName: "kube-api-access-xtqkp") pod "d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" (UID: "d8ab6a10-ee9d-45d9-9df7-5e92b2a91296"). InnerVolumeSpecName "kube-api-access-xtqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:34.948218 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:34.948180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:35.025192 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.025164 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:35.025192 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.025187 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqkp\" (UniqueName: \"kubernetes.io/projected/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296-kube-api-access-xtqkp\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:35.255844 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.255807 2559 generic.go:356] "Generic (PLEG): container finished" podID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" containerID="894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91" exitCode=0 Sep 16 13:56:35.256075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.255887 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" Sep 16 13:56:35.256075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.255893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" event={"ID":"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296","Type":"ContainerDied","Data":"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91"} Sep 16 13:56:35.256075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.255951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp" event={"ID":"d8ab6a10-ee9d-45d9-9df7-5e92b2a91296","Type":"ContainerDied","Data":"cc06417a44026dc2ffcf9a82d63964ee23964a3de92231e673ae6b853822fe03"} Sep 16 13:56:35.256075 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.255968 2559 scope.go:117] "RemoveContainer" containerID="894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91" Sep 16 13:56:35.265318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.265301 2559 scope.go:117] "RemoveContainer" containerID="894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91" Sep 16 13:56:35.265609 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:56:35.265591 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91\": container with ID starting with 894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91 not found: ID does not exist" containerID="894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91" Sep 16 13:56:35.265671 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.265618 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91"} err="failed to get container status \"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91\": rpc error: code = NotFound desc = could not find container \"894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91\": container with ID starting with 894db1241c52c852f578a240f6ccc5ad4241d5344410b347a0c011ef37bb2d91 not found: ID does not exist" Sep 16 13:56:35.283279 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.283252 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:35.286091 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.286058 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcc64e3-5797999754-4b2vp"] Sep 16 13:56:35.446118 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.446077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:35.471671 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.471566 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="143e9d4f-0d60-4766-9edb-9c2dfb4b8396" path="/var/lib/kubelet/pods/143e9d4f-0d60-4766-9edb-9c2dfb4b8396/volumes" Sep 16 13:56:35.472100 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.472085 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" path="/var/lib/kubelet/pods/d8ab6a10-ee9d-45d9-9df7-5e92b2a91296/volumes" Sep 16 13:56:35.585432 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.585343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:35.701484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.701441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:35.990503 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.990464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:56:35.990949 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.990931 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" containerName="payload-pre-processing" Sep 16 13:56:35.990949 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.990950 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" containerName="payload-pre-processing" Sep 16 13:56:35.991089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.990961 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="143e9d4f-0d60-4766-9edb-9c2dfb4b8396" containerName="revoke-keys" Sep 16 13:56:35.991089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.990967 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="143e9d4f-0d60-4766-9edb-9c2dfb4b8396" containerName="revoke-keys" Sep 16 13:56:35.991089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.991050 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ab6a10-ee9d-45d9-9df7-5e92b2a91296" containerName="payload-pre-processing" Sep 16 13:56:35.991089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.991061 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="143e9d4f-0d60-4766-9edb-9c2dfb4b8396" containerName="revoke-keys" Sep 16 13:56:35.994071 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.994052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:35.997764 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:35.997747 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ce782c2e-gw-openshift-default-dockercfg-vmdpz\"" Sep 16 13:56:36.042092 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.042055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:36.112189 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.112150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:56:36.136212 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136212 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7t8\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136484 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136833 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.136833 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.136516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.237977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.237936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.237977 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.237978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238208 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238208 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7t8\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238421 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238421 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238421 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238702 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238770 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238770 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.238898 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.238855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.240494 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.240469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.240740 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.240696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.248198 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.248168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.248326 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.248304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7t8\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8\") pod \"e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.254637 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.254586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:36.304063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.304022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:36.353261 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.352809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:36.582884 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.582842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:56:36.584527 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:36.584488 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82bcecb_a6a7_4b4f_a52a_bd62f92a2f3c.slice/crio-c7b8682fb7786915bd098e303687abe6eda7af26c5510b91b214cddf12a552e7 WatchSource:0}: Error finding container c7b8682fb7786915bd098e303687abe6eda7af26c5510b91b214cddf12a552e7: Status 404 returned error can't find the container with id c7b8682fb7786915bd098e303687abe6eda7af26c5510b91b214cddf12a552e7 Sep 16 13:56:36.586882 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.586839 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:36.587154 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.586925 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:36.587154 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:36.586960 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:37.200041 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.200003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:37.269059 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.269025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerStarted","Data":"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e"} Sep 16 13:56:37.269248 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.269090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerStarted","Data":"c7b8682fb7786915bd098e303687abe6eda7af26c5510b91b214cddf12a552e7"} Sep 16 13:56:37.301702 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.301644 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podStartSLOduration=2.30162741 podStartE2EDuration="2.30162741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:37.297030839 +0000 UTC m=+816.337624787" watchObservedRunningTime="2026-09-16 13:56:37.30162741 +0000 UTC m=+816.342221366" Sep 16 13:56:37.305097 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.305071 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:37.310452 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:37.310427 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:38.177044 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:38.177003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:38.274127 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:38.274086 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:38.275215 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:38.275194 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:56:39.052490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:39.052456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:39.334627 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:39.334520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:39.450742 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:39.450700 2559 patch_prober.go:28] interesting pod/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 16 13:56:39.450937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:39.450790 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:56:40.158753 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.158727 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:40.216316 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.216262 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:40.279921 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.279883 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.279935 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.279971 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwbhr\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280133 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280217 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280234 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:40.280323 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280257 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy\") pod \"eabf4894-4622-4581-8525-7da70abcfa06\" (UID: \"eabf4894-4622-4581-8525-7da70abcfa06\") " Sep 16 13:56:40.280558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280484 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:40.280614 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data" (OuterVolumeSpecName: "istio-data") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:40.280614 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280571 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.280614 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280573 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:40.280761 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.280730 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:40.282567 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.282523 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:40.282676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.282597 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr" (OuterVolumeSpecName: "kube-api-access-mwbhr") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "kube-api-access-mwbhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:40.282809 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.282793 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token" (OuterVolumeSpecName: "istio-token") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:40.282850 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.282834 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eabf4894-4622-4581-8525-7da70abcfa06" (UID: "eabf4894-4622-4581-8525-7da70abcfa06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:56:40.287368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.287345 2559 generic.go:356] "Generic (PLEG): container finished" podID="eabf4894-4622-4581-8525-7da70abcfa06" containerID="8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1" exitCode=0 Sep 16 13:56:40.287461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.287401 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" Sep 16 13:56:40.287461 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.287420 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerDied","Data":"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1"} Sep 16 13:56:40.287552 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.287482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk" event={"ID":"eabf4894-4622-4581-8525-7da70abcfa06","Type":"ContainerDied","Data":"8495c48625ffed886e7aa1765e1b3e14379574c9c057cacf59f7314a87237afc"} Sep 16 13:56:40.287552 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.287513 2559 scope.go:117] "RemoveContainer" containerID="8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1" Sep 16 13:56:40.307544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.307526 2559 scope.go:117] "RemoveContainer" containerID="aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b" Sep 16 13:56:40.316733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.316712 2559 scope.go:117] "RemoveContainer" containerID="8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1" Sep 16 13:56:40.317109 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:56:40.317088 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1\": container with ID starting with 8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1 not found: ID does not exist" containerID="8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1" Sep 16 13:56:40.317173 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.317118 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1"} err="failed to get container status \"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1\": rpc error: code = NotFound desc = could not find container \"8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1\": container with ID starting with 8133ed2b35eba2f893b704a1495b45b4a5660f6270224b811fd38be0e78c00b1 not found: ID does not exist" Sep 16 13:56:40.317173 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.317138 2559 scope.go:117] "RemoveContainer" containerID="aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b" Sep 16 13:56:40.317463 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:56:40.317442 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b\": container with ID starting with aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b not found: ID does not exist" containerID="aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b" Sep 16 13:56:40.317517 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.317472 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b"} err="failed to get container status \"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b\": rpc error: code = NotFound desc = could not find container \"aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b\": container with ID starting with aa299c58f8cb9874b656edbd22e66f73698a4bec83d949f86837885a5f0b6b5b not found: ID does not exist" Sep 16 13:56:40.318391 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.318371 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:56:40.322707 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.322685 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1dcc64e3-openshift-default-7b9cc4888b-hx2wk"] Sep 16 13:56:40.381679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381645 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbhr\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-kube-api-access-mwbhr\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.381679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381676 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.381679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381687 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabf4894-4622-4581-8525-7da70abcfa06-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.382200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381696 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.382200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381707 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabf4894-4622-4581-8525-7da70abcfa06-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.382200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381714 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.382200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381723 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabf4894-4622-4581-8525-7da70abcfa06-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:40.382200 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:40.381731 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabf4894-4622-4581-8525-7da70abcfa06-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:56:41.039479 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.039439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.141482 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.141451 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.240707 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.240667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.471454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.471420 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabf4894-4622-4581-8525-7da70abcfa06" path="/var/lib/kubelet/pods/eabf4894-4622-4581-8525-7da70abcfa06/volumes" Sep 16 13:56:41.569515 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.569478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:56:41.653005 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.652969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.761489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.761403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.769687 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.769653 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:56:41.770307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770287 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.770369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770311 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.770369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770328 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.770369 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770337 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.770464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770443 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.770464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.770456 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabf4894-4622-4581-8525-7da70abcfa06" containerName="istio-proxy" Sep 16 13:56:41.774846 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.774828 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:56:41.774964 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.774947 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:41.777893 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.777856 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ce782c2e\"" Sep 16 13:56:41.778462 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.778440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:41.786439 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.786407 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:56:41.789206 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.789182 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:56:41.897728 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.897680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:41.897941 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.897821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhvw\" (UniqueName: \"kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:41.897941 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.897934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:41.898084 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.897978 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phdz\" (UniqueName: \"kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:41.999476 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.999440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:41.999659 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.999483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phdz\" (UniqueName: \"kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:41.999659 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.999603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:41.999788 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:41.999698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhvw\" (UniqueName: \"kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:42.000220 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.000194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:42.000418 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.000398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:42.009131 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.009109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhvw\" (UniqueName: \"kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw\") pod \"payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:42.009330 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.009308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phdz\" (UniqueName: \"kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz\") pod \"payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:42.090306 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.090215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:42.096670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.096645 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:42.292334 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.292300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:42.365311 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.365278 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:56:42.367143 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:42.367109 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aedb126_7dd8_413b_966c_d4c1bd7b1cbf.slice/crio-bf2a32e922401bcea42427f2d0380f1c879417c5163750a1a9c64e88062da2bc WatchSource:0}: Error finding container bf2a32e922401bcea42427f2d0380f1c879417c5163750a1a9c64e88062da2bc: Status 404 returned error can't find the container with id bf2a32e922401bcea42427f2d0380f1c879417c5163750a1a9c64e88062da2bc Sep 16 13:56:42.381949 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:42.381926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:56:42.383752 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:42.383719 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5ba9c0_283a_4853_9630_06fa71603a6e.slice/crio-99e09d1f3286ac39ae3d1290a5a2f35c3c8511cc19056d07198e28456ca4a9c8 WatchSource:0}: Error finding container 99e09d1f3286ac39ae3d1290a5a2f35c3c8511cc19056d07198e28456ca4a9c8: Status 404 returned error can't find the container with id 99e09d1f3286ac39ae3d1290a5a2f35c3c8511cc19056d07198e28456ca4a9c8 Sep 16 13:56:43.303727 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.303686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" event={"ID":"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf","Type":"ContainerStarted","Data":"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af"} Sep 16 13:56:43.304271 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.303759 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" event={"ID":"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf","Type":"ContainerStarted","Data":"bf2a32e922401bcea42427f2d0380f1c879417c5163750a1a9c64e88062da2bc"} Sep 16 13:56:43.304271 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.303790 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:43.305417 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.305388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" event={"ID":"9d5ba9c0-283a-4853-9630-06fa71603a6e","Type":"ContainerStarted","Data":"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31"} Sep 16 13:56:43.305564 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.305445 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:43.305564 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.305467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" event={"ID":"9d5ba9c0-283a-4853-9630-06fa71603a6e","Type":"ContainerStarted","Data":"99e09d1f3286ac39ae3d1290a5a2f35c3c8511cc19056d07198e28456ca4a9c8"} Sep 16 13:56:43.324210 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.324131 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" podStartSLOduration=2.324111875 podStartE2EDuration="2.324111875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:43.323244702 +0000 UTC m=+822.363838657" watchObservedRunningTime="2026-09-16 13:56:43.324111875 +0000 UTC m=+822.364705830" Sep 16 13:56:43.343569 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.343512 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" podStartSLOduration=2.3434917840000002 podStartE2EDuration="2.343491784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:43.342200782 +0000 UTC m=+822.382794739" watchObservedRunningTime="2026-09-16 13:56:43.343491784 +0000 UTC m=+822.384085740" Sep 16 13:56:43.392022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:43.391980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:44.267433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:44.267388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:45.201088 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:45.201049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:46.530033 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.529997 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 13:56:46.533880 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.533852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.538141 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.538118 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"kube-root-ca.crt\"" Sep 16 13:56:46.538993 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.538970 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"simulator-w4-19203d-kserve-self-signed-certs\"" Sep 16 13:56:46.538993 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.538979 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"default-dockercfg-lcbzq\"" Sep 16 13:56:46.538993 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.538992 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"openshift-service-ca.crt\"" Sep 16 13:56:46.545457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.545436 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 13:56:46.646395 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646361 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.646558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.646558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.646558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646481 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.646558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646507 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.646558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.646537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcfg\" (UniqueName: \"kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.747791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.747791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcfg\" (UniqueName: \"kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748063 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.747958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.748259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748352 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.748301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.748352 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.748327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.750223 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.750200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.750408 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.750390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.756720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.756698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcfg\" (UniqueName: \"kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg\") pod \"simulator-w4-19203d-kserve-7cc98d9996-x4zqs\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:46.844130 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:46.844032 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:47.123543 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:47.123430 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 13:56:47.126151 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:56:47.126122 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda241c8ef_0ced_41ce_acfd_e5c34c7dc2ff.slice/crio-7296fe8dc18be42c38180fce161d6e16c6d2aea88c200831a00736181852c803 WatchSource:0}: Error finding container 7296fe8dc18be42c38180fce161d6e16c6d2aea88c200831a00736181852c803: Status 404 returned error can't find the container with id 7296fe8dc18be42c38180fce161d6e16c6d2aea88c200831a00736181852c803 Sep 16 13:56:47.154292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:47.154266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:47.327693 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:47.327659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" event={"ID":"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff","Type":"ContainerStarted","Data":"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702"} Sep 16 13:56:47.327918 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:47.327709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" event={"ID":"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff","Type":"ContainerStarted","Data":"7296fe8dc18be42c38180fce161d6e16c6d2aea88c200831a00736181852c803"} Sep 16 13:56:47.349624 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:47.349571 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" podStartSLOduration=1.349555063 podStartE2EDuration="1.349555063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:47.347037423 +0000 UTC m=+826.387631377" watchObservedRunningTime="2026-09-16 13:56:47.349555063 +0000 UTC m=+826.390149055" Sep 16 13:56:54.312615 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:54.312578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:56:54.313003 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:54.312741 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:56:56.844998 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:56.844952 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:56.845390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:56.845027 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:56.858236 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:56.858205 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:57.356900 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:57.356844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:56:57.411077 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:57.411048 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 13:56:59.411381 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.411346 2559 generic.go:356] "Generic (PLEG): container finished" podID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerID="5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e" exitCode=0 Sep 16 13:56:59.411905 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.411418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerDied","Data":"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e"} Sep 16 13:56:59.411905 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.411837 2559 scope.go:117] "RemoveContainer" containerID="5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e" Sep 16 13:56:59.412986 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.412955 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:59.413072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.413019 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:56:59.413072 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:56:59.413052 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:57:00.424681 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:00.424642 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerStarted","Data":"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797"} Sep 16 13:57:01.304199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:01.304159 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:01.304511 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:01.304485 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:57:01.304578 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:01.304537 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 16 13:57:02.304986 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:02.304945 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:57:02.305433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:02.305010 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 16 13:57:03.304911 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:03.304846 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:57:03.305122 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:03.304940 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 16 13:57:03.587676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:03.587590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:04.304795 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:04.304757 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:57:04.305014 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:04.304816 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 16 13:57:04.879599 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:04.879559 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:05.304638 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.304601 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:57:05.304812 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.304657 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 16 13:57:05.412175 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.412130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:05.443145 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.443111 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 13:57:05.449338 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.449314 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.451620 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.451599 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"openshift-service-ca.crt\"" Sep 16 13:57:05.452489 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.452464 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"kube-root-ca.crt\"" Sep 16 13:57:05.452591 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.452465 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"default-dockercfg-vk8k8\"" Sep 16 13:57:05.452591 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.452577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"simulator-w0-035c64-kserve-self-signed-certs\"" Sep 16 13:57:05.455846 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.455819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 13:57:05.520294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520261 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j696\" (UniqueName: \"kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.520464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520308 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.520464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.520464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.520464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.520464 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.520447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.621782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.621662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.621782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.621706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.621782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.621736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.621782 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.621763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.622271 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.621843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j696\" (UniqueName: \"kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.622501 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.622472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.622616 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.622251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.622616 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.622301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.622616 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.622220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.624655 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.624621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.625026 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.625002 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.633147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.633122 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j696\" (UniqueName: \"kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696\") pod \"simulator-w0-035c64-kserve-758fc5cc9b-rg4zr\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:05.758687 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.758650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:05.759626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:05.759602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:06.034662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.034631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 13:57:06.036966 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:57:06.036936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0314deaf_0aed_447e_b279_222e6919dd81.slice/crio-5dfb3227c2675a5e6600eb13ed85968eb0f15f5c3692c7560ba2051add0860ea WatchSource:0}: Error finding container 5dfb3227c2675a5e6600eb13ed85968eb0f15f5c3692c7560ba2051add0860ea: Status 404 returned error can't find the container with id 5dfb3227c2675a5e6600eb13ed85968eb0f15f5c3692c7560ba2051add0860ea Sep 16 13:57:06.287193 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.287105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:06.308491 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.308461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:06.308883 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.308854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:06.309502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.309486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:06.456767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.456723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" event={"ID":"0314deaf-0aed-447e-b279-222e6919dd81","Type":"ContainerStarted","Data":"6cf7abfd01e8235829e0c383c7f2a45f75eb32789e159a851283706432f7ceac"} Sep 16 13:57:06.456767 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.456775 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" event={"ID":"0314deaf-0aed-447e-b279-222e6919dd81","Type":"ContainerStarted","Data":"5dfb3227c2675a5e6600eb13ed85968eb0f15f5c3692c7560ba2051add0860ea"} Sep 16 13:57:06.476601 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:06.476552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" podStartSLOduration=1.476537758 podStartE2EDuration="1.476537758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:06.474629633 +0000 UTC m=+845.515223589" watchObservedRunningTime="2026-09-16 13:57:06.476537758 +0000 UTC m=+845.517131712" Sep 16 13:57:07.073251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:07.073212 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:07.954060 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:07.954024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:08.831105 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:08.831070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:09.689604 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:09.689573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.059571 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:11.059530 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:57:15.760714 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:15.760676 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:15.761328 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:15.760805 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:15.773312 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:15.773284 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:16.515812 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:16.515778 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 13:57:20.070393 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.070349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.828294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.828263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.859305 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.859274 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:57:20.859503 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.859477 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" podUID="9d5ba9c0-283a-4853-9630-06fa71603a6e" containerName="payload-processing" containerID="cri-o://b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31" gracePeriod=30 Sep 16 13:57:20.865765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.865738 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:57:20.867176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.867146 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" podUID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" containerName="payload-pre-processing" containerID="cri-o://0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af" gracePeriod=30 Sep 16 13:57:20.910346 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:20.910317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.010987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.010942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.109171 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.109142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.119481 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.119459 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:57:21.142089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.142054 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:57:21.168749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.168720 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume\") pod \"9d5ba9c0-283a-4853-9630-06fa71603a6e\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " Sep 16 13:57:21.168946 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.168900 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4phdz\" (UniqueName: \"kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz\") pod \"9d5ba9c0-283a-4853-9630-06fa71603a6e\" (UID: \"9d5ba9c0-283a-4853-9630-06fa71603a6e\") " Sep 16 13:57:21.169164 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.169136 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d5ba9c0-283a-4853-9630-06fa71603a6e" (UID: "9d5ba9c0-283a-4853-9630-06fa71603a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:21.171033 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.171009 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz" (OuterVolumeSpecName: "kube-api-access-4phdz") pod "9d5ba9c0-283a-4853-9630-06fa71603a6e" (UID: "9d5ba9c0-283a-4853-9630-06fa71603a6e"). InnerVolumeSpecName "kube-api-access-4phdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:21.213127 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.213091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.227436 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.227405 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:57:21.227679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.227654 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" containerID="cri-o://d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797" gracePeriod=30 Sep 16 13:57:21.270253 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.270218 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkhvw\" (UniqueName: \"kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw\") pod \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " Sep 16 13:57:21.270447 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.270274 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume\") pod \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\" (UID: \"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf\") " Sep 16 13:57:21.270567 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.270546 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ba9c0-283a-4853-9630-06fa71603a6e-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.270567 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.270563 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phdz\" (UniqueName: \"kubernetes.io/projected/9d5ba9c0-283a-4853-9630-06fa71603a6e-kube-api-access-4phdz\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.270713 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.270676 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" (UID: "7aedb126-7dd8-413b-966c-d4c1bd7b1cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:21.272267 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.272242 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw" (OuterVolumeSpecName: "kube-api-access-qkhvw") pod "7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" (UID: "7aedb126-7dd8-413b-966c-d4c1bd7b1cbf"). InnerVolumeSpecName "kube-api-access-qkhvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:21.289549 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.289522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.311843 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.311804 2559 patch_prober.go:28] interesting pod/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:57:21.312014 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.311899 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:57:21.371360 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.371276 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkhvw\" (UniqueName: \"kubernetes.io/projected/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-kube-api-access-qkhvw\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.371360 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.371303 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.414453 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.414421 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:21.530137 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.530103 2559 generic.go:356] "Generic (PLEG): container finished" podID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" containerID="0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af" exitCode=0 Sep 16 13:57:21.530318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.530149 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" Sep 16 13:57:21.530318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.530179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" event={"ID":"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf","Type":"ContainerDied","Data":"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af"} Sep 16 13:57:21.530318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.530227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q" event={"ID":"7aedb126-7dd8-413b-966c-d4c1bd7b1cbf","Type":"ContainerDied","Data":"bf2a32e922401bcea42427f2d0380f1c879417c5163750a1a9c64e88062da2bc"} Sep 16 13:57:21.530318 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.530244 2559 scope.go:117] "RemoveContainer" containerID="0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af" Sep 16 13:57:21.531568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.531545 2559 generic.go:356] "Generic (PLEG): container finished" podID="9d5ba9c0-283a-4853-9630-06fa71603a6e" containerID="b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31" exitCode=0 Sep 16 13:57:21.531682 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.531585 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" Sep 16 13:57:21.531682 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.531630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" event={"ID":"9d5ba9c0-283a-4853-9630-06fa71603a6e","Type":"ContainerDied","Data":"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31"} Sep 16 13:57:21.531682 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.531678 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk" event={"ID":"9d5ba9c0-283a-4853-9630-06fa71603a6e","Type":"ContainerDied","Data":"99e09d1f3286ac39ae3d1290a5a2f35c3c8511cc19056d07198e28456ca4a9c8"} Sep 16 13:57:21.539281 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.539262 2559 scope.go:117] "RemoveContainer" containerID="0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af" Sep 16 13:57:21.539584 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:57:21.539566 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af\": container with ID starting with 0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af not found: ID does not exist" containerID="0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af" Sep 16 13:57:21.539632 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.539592 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af"} err="failed to get container status \"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af\": rpc error: code = NotFound desc = could not find container \"0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af\": container with ID starting with 0bd10a5f039df1656d74d79a49e1239e088f51bfb7888cfb7168d543def782af not found: ID does not exist" Sep 16 13:57:21.539632 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.539609 2559 scope.go:117] "RemoveContainer" containerID="b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31" Sep 16 13:57:21.547383 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.547366 2559 scope.go:117] "RemoveContainer" containerID="b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31" Sep 16 13:57:21.547640 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:57:21.547622 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31\": container with ID starting with b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31 not found: ID does not exist" containerID="b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31" Sep 16 13:57:21.547681 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.547646 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31"} err="failed to get container status \"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31\": rpc error: code = NotFound desc = could not find container \"b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31\": container with ID starting with b347dfa885d47e4652b0d462d8e90011d35e4a551e01b3de51d7addf71f0df31 not found: ID does not exist" Sep 16 13:57:21.551400 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.551381 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:57:21.557108 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.557079 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ce782c2e-94665d5c4-45v4q"] Sep 16 13:57:21.571817 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.571796 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:57:21.579584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.579564 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ce782c2e-665c5585f8-t5xwk"] Sep 16 13:57:21.707854 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:21.707827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:22.033557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:22.033481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:22.398660 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:22.398547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:22.545024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:22.544992 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:23.278630 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:23.278594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:23.471264 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:23.471233 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" path="/var/lib/kubelet/pods/7aedb126-7dd8-413b-966c-d4c1bd7b1cbf/volumes" Sep 16 13:57:23.471622 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:23.471598 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5ba9c0-283a-4853-9630-06fa71603a6e" path="/var/lib/kubelet/pods/9d5ba9c0-283a-4853-9630-06fa71603a6e/volumes" Sep 16 13:57:24.203975 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:24.203946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:24.388719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:24.388687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:25.267050 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:25.267021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:26.246987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.246953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:26.468901 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.468861 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:26.520750 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.520677 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.520892 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.520750 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.520892 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.520795 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.520892 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.520830 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521062 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.520956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db7t8\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521124 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521066 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521176 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521245 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521302 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521133 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:26.521302 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521211 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:26.521302 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521266 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy\") pod \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\" (UID: \"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c\") " Sep 16 13:57:26.521444 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521328 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:26.521496 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521433 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:26.521496 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521460 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:26.521717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521626 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.521717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521650 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.521717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521665 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.521717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521680 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.521717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.521694 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.523167 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.523143 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:26.523292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.523265 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8" (OuterVolumeSpecName: "kube-api-access-db7t8") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "kube-api-access-db7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:26.523379 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.523361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:57:26.523455 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.523435 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" (UID: "a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:26.563329 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.563296 2559 generic.go:356] "Generic (PLEG): container finished" podID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerID="d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797" exitCode=0 Sep 16 13:57:26.563502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.563356 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" Sep 16 13:57:26.563502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.563382 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerDied","Data":"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797"} Sep 16 13:57:26.563502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.563442 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9" event={"ID":"a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c","Type":"ContainerDied","Data":"c7b8682fb7786915bd098e303687abe6eda7af26c5510b91b214cddf12a552e7"} Sep 16 13:57:26.563502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.563466 2559 scope.go:117] "RemoveContainer" containerID="d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797" Sep 16 13:57:26.574804 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.574779 2559 scope.go:117] "RemoveContainer" containerID="5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e" Sep 16 13:57:26.583847 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.583827 2559 scope.go:117] "RemoveContainer" containerID="d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797" Sep 16 13:57:26.584159 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:57:26.584143 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797\": container with ID starting with d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797 not found: ID does not exist" containerID="d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797" Sep 16 13:57:26.584222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.584167 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797"} err="failed to get container status \"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797\": rpc error: code = NotFound desc = could not find container \"d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797\": container with ID starting with d35285a4f27e72783f205f2f4ff072057d0f998d1fb51235cf28255a8674d797 not found: ID does not exist" Sep 16 13:57:26.584222 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.584182 2559 scope.go:117] "RemoveContainer" containerID="5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e" Sep 16 13:57:26.584459 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:57:26.584446 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e\": container with ID starting with 5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e not found: ID does not exist" containerID="5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e" Sep 16 13:57:26.584515 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.584461 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e"} err="failed to get container status \"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e\": rpc error: code = NotFound desc = could not find container \"5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e\": container with ID starting with 5c621ddc6e1aeb7502617850945b722f1240b2e8070556564c04d12851be440e not found: ID does not exist" Sep 16 13:57:26.585319 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.585298 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:57:26.594089 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.594063 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:57:26.602234 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.602211 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ce782c2e-gw-openshift-default-79776bf84-njqv9"] Sep 16 13:57:26.622319 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.622291 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.622319 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.622318 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.622487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.622330 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:26.622487 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:26.622340 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7t8\" (UniqueName: \"kubernetes.io/projected/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c-kube-api-access-db7t8\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:57:27.333312 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:27.333274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:27.434257 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:27.434222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:27.471746 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:27.471710 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" path="/var/lib/kubelet/pods/a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c/volumes" Sep 16 13:57:27.740225 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:27.740187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:27.896783 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:27.896747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.011934 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.011828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.061287 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061230 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:57:28.061626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061614 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" containerName="payload-pre-processing" Sep 16 13:57:28.061670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061628 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" containerName="payload-pre-processing" Sep 16 13:57:28.061670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061637 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5ba9c0-283a-4853-9630-06fa71603a6e" containerName="payload-processing" Sep 16 13:57:28.061670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061644 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5ba9c0-283a-4853-9630-06fa71603a6e" containerName="payload-processing" Sep 16 13:57:28.061670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061652 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061658 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061672 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061678 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061734 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061744 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aedb126-7dd8-413b-966c-d4c1bd7b1cbf" containerName="payload-pre-processing" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061750 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82bcecb-a6a7-4b4f-a52a-bd62f92a2f3c" containerName="istio-proxy" Sep 16 13:57:28.061891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.061757 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5ba9c0-283a-4853-9630-06fa71603a6e" containerName="payload-processing" Sep 16 13:57:28.064714 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.064694 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.067538 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.067516 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-38e8bf38\"" Sep 16 13:57:28.080888 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.080839 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:57:28.135658 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.135628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.135818 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.135694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rfk\" (UniqueName: \"kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.140334 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.140306 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:57:28.143732 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.143706 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.147164 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.147149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-38e8bf38-dockercfg-z2fp4\"" Sep 16 13:57:28.157712 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.157690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:57:28.210173 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.210141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.236403 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.236368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.236588 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.236453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rfk\" (UniqueName: \"kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.236588 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.236495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xcrk\" (UniqueName: \"kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.236588 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.236531 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.237133 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.237113 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.246474 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.246444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rfk\" (UniqueName: \"kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk\") pod \"payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.337389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.337295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.337572 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.337407 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xcrk\" (UniqueName: \"kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.338007 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.337986 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.347186 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.347159 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xcrk\" (UniqueName: \"kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk\") pod \"payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.375054 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.375026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:28.454179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.454146 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:28.496465 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.496430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.655304 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.655275 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:57:28.656285 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:57:28.656260 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc186099e_1efe_404c_a8cc_3b42ff57122b.slice/crio-a5da576af70869c8c29fe5045040586dee21dfa2bb2816b283128d068e00874e WatchSource:0}: Error finding container a5da576af70869c8c29fe5045040586dee21dfa2bb2816b283128d068e00874e: Status 404 returned error can't find the container with id a5da576af70869c8c29fe5045040586dee21dfa2bb2816b283128d068e00874e Sep 16 13:57:28.733641 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:28.733614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:57:28.735133 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:57:28.735106 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750f9777_a6bc_4463_962e_b064d19f3271.slice/crio-b90a954b54a5688014b67f5e35369afa8d7ebe1fbc2c16585416996b779206d0 WatchSource:0}: Error finding container b90a954b54a5688014b67f5e35369afa8d7ebe1fbc2c16585416996b779206d0: Status 404 returned error can't find the container with id b90a954b54a5688014b67f5e35369afa8d7ebe1fbc2c16585416996b779206d0 Sep 16 13:57:29.376006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.375976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:29.582067 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.582031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" event={"ID":"c186099e-1efe-404c-a8cc-3b42ff57122b","Type":"ContainerStarted","Data":"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519"} Sep 16 13:57:29.582519 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.582098 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:29.582519 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.582123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" event={"ID":"c186099e-1efe-404c-a8cc-3b42ff57122b","Type":"ContainerStarted","Data":"a5da576af70869c8c29fe5045040586dee21dfa2bb2816b283128d068e00874e"} Sep 16 13:57:29.583686 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.583658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" event={"ID":"750f9777-a6bc-4463-962e-b064d19f3271","Type":"ContainerStarted","Data":"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c"} Sep 16 13:57:29.583833 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.583702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" event={"ID":"750f9777-a6bc-4463-962e-b064d19f3271","Type":"ContainerStarted","Data":"b90a954b54a5688014b67f5e35369afa8d7ebe1fbc2c16585416996b779206d0"} Sep 16 13:57:29.583833 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.583723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:29.602479 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.602431 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" podStartSLOduration=1.602415708 podStartE2EDuration="1.602415708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:29.600761294 +0000 UTC m=+868.641355253" watchObservedRunningTime="2026-09-16 13:57:29.602415708 +0000 UTC m=+868.643009710" Sep 16 13:57:29.620496 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:29.620443 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" podStartSLOduration=1.6204268389999998 podStartE2EDuration="1.620426839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:29.619155241 +0000 UTC m=+868.659749199" watchObservedRunningTime="2026-09-16 13:57:29.620426839 +0000 UTC m=+868.661020793" Sep 16 13:57:30.190468 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:30.190433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:31.101674 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:31.101634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:32.434760 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.434725 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:57:32.438490 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.438471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.442022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.441997 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-617597\"/\"openshift-service-ca.crt\"" Sep 16 13:57:32.442147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.442058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-617597\"/\"default-dockercfg-tmg7b\"" Sep 16 13:57:32.442147 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.442137 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-617597\"/\"autores-model-617597-kserve-self-signed-certs\"" Sep 16 13:57:32.442325 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.442312 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-617597\"/\"kube-root-ca.crt\"" Sep 16 13:57:32.449557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.449531 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:57:32.579735 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.579692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.579735 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.579738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.580025 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.579822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.580151 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.580127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2gz\" (UniqueName: \"kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.580240 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.580190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.580240 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.580227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681272 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2gz\" (UniqueName: \"kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681429 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681666 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681733 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681778 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681741 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.681778 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.681754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.683558 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.683535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.683827 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.683811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.690567 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.690346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2gz\" (UniqueName: \"kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz\") pod \"autores-model-617597-kserve-57d78cb5d7-6hbt2\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:32.737567 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.737530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:32.748551 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:32.748524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:33.021135 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:33.021110 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:57:33.022859 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:57:33.022831 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f1c36d6_5802_4d43_a7c3_30b060b93dc3.slice/crio-befe4d4e6c3caac5bef9fd2ca75f6a15c9e7b8bd92a64d3c7b98f9ed83e9120c WatchSource:0}: Error finding container befe4d4e6c3caac5bef9fd2ca75f6a15c9e7b8bd92a64d3c7b98f9ed83e9120c: Status 404 returned error can't find the container with id befe4d4e6c3caac5bef9fd2ca75f6a15c9e7b8bd92a64d3c7b98f9ed83e9120c Sep 16 13:57:33.606617 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:33.606576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" event={"ID":"7f1c36d6-5802-4d43-a7c3-30b060b93dc3","Type":"ContainerStarted","Data":"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67"} Sep 16 13:57:33.606617 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:33.606625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" event={"ID":"7f1c36d6-5802-4d43-a7c3-30b060b93dc3","Type":"ContainerStarted","Data":"befe4d4e6c3caac5bef9fd2ca75f6a15c9e7b8bd92a64d3c7b98f9ed83e9120c"} Sep 16 13:57:33.628704 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:33.628651 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" podStartSLOduration=1.628635811 podStartE2EDuration="1.628635811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:33.625734273 +0000 UTC m=+872.666328227" watchObservedRunningTime="2026-09-16 13:57:33.628635811 +0000 UTC m=+872.669229765" Sep 16 13:57:40.591249 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:40.591216 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:57:40.591636 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:40.591289 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:57:42.749607 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:42.749569 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:42.750112 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:42.749650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:42.762185 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:42.762160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:43.672604 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:43.672577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:57:49.999022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:49.998975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:50.050050 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.050013 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 13:57:50.053405 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.053386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.056769 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.056743 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b52a89\"/\"kube-root-ca.crt\"" Sep 16 13:57:50.056769 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.056757 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b52a89\"/\"test-model-b52a89-kserve-self-signed-certs\"" Sep 16 13:57:50.056769 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.056760 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b52a89\"/\"default-dockercfg-86kw7\"" Sep 16 13:57:50.057002 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.056761 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b52a89\"/\"openshift-service-ca.crt\"" Sep 16 13:57:50.067613 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.067592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 13:57:50.142897 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.142849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.143012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.142922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.143012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.142950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.143012 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.142986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ptv\" (UniqueName: \"kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.143119 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.143055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.143119 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.143085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.173435 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.173403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:50.243759 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.243705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.243771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.243931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244159 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ptv\" (UniqueName: \"kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244159 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244159 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244549 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244640 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.244748 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.244702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.246324 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.246303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.246598 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.246580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.253994 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.253944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ptv\" (UniqueName: \"kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv\") pod \"test-model-b52a89-kserve-85dfb984bc-xpdr8\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.389481 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.389436 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:57:50.662106 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.662077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 13:57:50.663705 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:57:50.663676 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25f0014b_37c2_4084_804f_33f62f74a7bc.slice/crio-e0f33904a83c8ef140f71ca4d841395f2639c13ed7b484074614ccbaa5376bcc WatchSource:0}: Error finding container e0f33904a83c8ef140f71ca4d841395f2639c13ed7b484074614ccbaa5376bcc: Status 404 returned error can't find the container with id e0f33904a83c8ef140f71ca4d841395f2639c13ed7b484074614ccbaa5376bcc Sep 16 13:57:50.700074 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.700045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" event={"ID":"25f0014b-37c2-4084-804f-33f62f74a7bc","Type":"ContainerStarted","Data":"e0f33904a83c8ef140f71ca4d841395f2639c13ed7b484074614ccbaa5376bcc"} Sep 16 13:57:50.978197 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:50.978161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:51.706374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:51.706333 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" event={"ID":"25f0014b-37c2-4084-804f-33f62f74a7bc","Type":"ContainerStarted","Data":"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316"} Sep 16 13:57:51.725698 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:51.725646 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" podStartSLOduration=1.7256291369999999 podStartE2EDuration="1.725629137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:51.723721961 +0000 UTC m=+890.764315916" watchObservedRunningTime="2026-09-16 13:57:51.725629137 +0000 UTC m=+890.766223090" Sep 16 13:57:51.978178 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:51.978084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:52.900227 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:52.900184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:53.773895 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:53.773789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:54.355405 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:54.355368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:57:54.714006 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:54.713966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:55.653587 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:55.653549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:56.646164 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:56.646127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:57.586394 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:57.586334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:58.402694 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:58.402661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:59.459637 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:57:59.459603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:00.389657 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:00.389615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:58:00.389846 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:00.389688 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:58:00.402307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:00.402279 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:58:00.418105 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:00.418077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:00.762193 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:00.762167 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 13:58:01.382973 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:01.382935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:04.510602 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:04.510562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:05.473628 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:05.473595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:06.292634 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:06.292594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:07.415808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:07.415776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:08.373199 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:08.373162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:09.835292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:09.835256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:10.287260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:10.287219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:10.563858 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:10.563772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:58:11.198307 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:11.198270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:12.036426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.036387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:12.080343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.080312 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:58:12.080551 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.080528 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" podUID="c186099e-1efe-404c-a8cc-3b42ff57122b" containerName="payload-processing" containerID="cri-o://db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519" gracePeriod=30 Sep 16 13:58:12.087451 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.087422 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:58:12.087719 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.087679 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" podUID="750f9777-a6bc-4463-962e-b064d19f3271" containerName="payload-pre-processing" containerID="cri-o://a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c" gracePeriod=30 Sep 16 13:58:12.354632 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.354608 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:58:12.357711 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.357690 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:58:12.435703 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.435675 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xcrk\" (UniqueName: \"kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk\") pod \"750f9777-a6bc-4463-962e-b064d19f3271\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " Sep 16 13:58:12.435703 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.435708 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume\") pod \"c186099e-1efe-404c-a8cc-3b42ff57122b\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " Sep 16 13:58:12.435937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.435749 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume\") pod \"750f9777-a6bc-4463-962e-b064d19f3271\" (UID: \"750f9777-a6bc-4463-962e-b064d19f3271\") " Sep 16 13:58:12.435937 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.435817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6rfk\" (UniqueName: \"kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk\") pod \"c186099e-1efe-404c-a8cc-3b42ff57122b\" (UID: \"c186099e-1efe-404c-a8cc-3b42ff57122b\") " Sep 16 13:58:12.436070 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.436049 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c186099e-1efe-404c-a8cc-3b42ff57122b" (UID: "c186099e-1efe-404c-a8cc-3b42ff57122b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:12.436163 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.436144 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c186099e-1efe-404c-a8cc-3b42ff57122b-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.436226 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.436147 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "750f9777-a6bc-4463-962e-b064d19f3271" (UID: "750f9777-a6bc-4463-962e-b064d19f3271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:12.437829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.437801 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk" (OuterVolumeSpecName: "kube-api-access-8xcrk") pod "750f9777-a6bc-4463-962e-b064d19f3271" (UID: "750f9777-a6bc-4463-962e-b064d19f3271"). InnerVolumeSpecName "kube-api-access-8xcrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:12.437829 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.437832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk" (OuterVolumeSpecName: "kube-api-access-f6rfk") pod "c186099e-1efe-404c-a8cc-3b42ff57122b" (UID: "c186099e-1efe-404c-a8cc-3b42ff57122b"). InnerVolumeSpecName "kube-api-access-f6rfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:12.536775 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.536742 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xcrk\" (UniqueName: \"kubernetes.io/projected/750f9777-a6bc-4463-962e-b064d19f3271-kube-api-access-8xcrk\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.536775 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.536771 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750f9777-a6bc-4463-962e-b064d19f3271-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.537090 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.536785 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rfk\" (UniqueName: \"kubernetes.io/projected/c186099e-1efe-404c-a8cc-3b42ff57122b-kube-api-access-f6rfk\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.814145 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.814105 2559 generic.go:356] "Generic (PLEG): container finished" podID="750f9777-a6bc-4463-962e-b064d19f3271" containerID="a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c" exitCode=0 Sep 16 13:58:12.814351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.814151 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" Sep 16 13:58:12.814351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.814188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" event={"ID":"750f9777-a6bc-4463-962e-b064d19f3271","Type":"ContainerDied","Data":"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c"} Sep 16 13:58:12.814351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.814249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4" event={"ID":"750f9777-a6bc-4463-962e-b064d19f3271","Type":"ContainerDied","Data":"b90a954b54a5688014b67f5e35369afa8d7ebe1fbc2c16585416996b779206d0"} Sep 16 13:58:12.814351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.814277 2559 scope.go:117] "RemoveContainer" containerID="a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c" Sep 16 13:58:12.815700 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.815676 2559 generic.go:356] "Generic (PLEG): container finished" podID="c186099e-1efe-404c-a8cc-3b42ff57122b" containerID="db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519" exitCode=0 Sep 16 13:58:12.815838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.815736 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" Sep 16 13:58:12.815838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.815799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" event={"ID":"c186099e-1efe-404c-a8cc-3b42ff57122b","Type":"ContainerDied","Data":"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519"} Sep 16 13:58:12.815971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.815841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm" event={"ID":"c186099e-1efe-404c-a8cc-3b42ff57122b","Type":"ContainerDied","Data":"a5da576af70869c8c29fe5045040586dee21dfa2bb2816b283128d068e00874e"} Sep 16 13:58:12.823717 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.823698 2559 scope.go:117] "RemoveContainer" containerID="a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c" Sep 16 13:58:12.824027 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:58:12.824010 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c\": container with ID starting with a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c not found: ID does not exist" containerID="a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c" Sep 16 13:58:12.824116 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.824034 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c"} err="failed to get container status \"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c\": rpc error: code = NotFound desc = could not find container \"a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c\": container with ID starting with a2705d185a746589ff8182eca5706add96fe6be34a15e5e55dadf790ede5d71c not found: ID does not exist" Sep 16 13:58:12.824116 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.824051 2559 scope.go:117] "RemoveContainer" containerID="db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519" Sep 16 13:58:12.832231 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.832215 2559 scope.go:117] "RemoveContainer" containerID="db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519" Sep 16 13:58:12.832457 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:58:12.832440 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519\": container with ID starting with db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519 not found: ID does not exist" containerID="db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519" Sep 16 13:58:12.832502 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.832463 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519"} err="failed to get container status \"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519\": rpc error: code = NotFound desc = could not find container \"db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519\": container with ID starting with db37dbdc328f3bbfc89708e4fdb3dcdc6618993e1014604f886fba3d3d522519 not found: ID does not exist" Sep 16 13:58:12.866280 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.866248 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:58:12.877759 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.877736 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38e8bf38-59b5fddbb9-jpwjm"] Sep 16 13:58:12.905891 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.905852 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:58:12.913195 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:12.913171 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38e8bf38-784565fdb6-t6sf4"] Sep 16 13:58:13.143209 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.143117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:13.257115 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.257084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:13.474639 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.474596 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750f9777-a6bc-4463-962e-b064d19f3271" path="/var/lib/kubelet/pods/750f9777-a6bc-4463-962e-b064d19f3271/volumes" Sep 16 13:58:13.475146 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.475124 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c186099e-1efe-404c-a8cc-3b42ff57122b" path="/var/lib/kubelet/pods/c186099e-1efe-404c-a8cc-3b42ff57122b/volumes" Sep 16 13:58:13.641001 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.640967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:13.993919 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:13.993863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.113355 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.113319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.400446 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.400365 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:58:14.401039 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401018 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c186099e-1efe-404c-a8cc-3b42ff57122b" containerName="payload-processing" Sep 16 13:58:14.401128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401043 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c186099e-1efe-404c-a8cc-3b42ff57122b" containerName="payload-processing" Sep 16 13:58:14.401128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401076 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750f9777-a6bc-4463-962e-b064d19f3271" containerName="payload-pre-processing" Sep 16 13:58:14.401128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401087 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="750f9777-a6bc-4463-962e-b064d19f3271" containerName="payload-pre-processing" Sep 16 13:58:14.401231 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401186 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c186099e-1efe-404c-a8cc-3b42ff57122b" containerName="payload-processing" Sep 16 13:58:14.401231 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.401202 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="750f9777-a6bc-4463-962e-b064d19f3271" containerName="payload-pre-processing" Sep 16 13:58:14.406238 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.406215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.409597 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.409576 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1852eb66-openshift-default-dockercfg-jstl5\"" Sep 16 13:58:14.427293 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.427256 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:58:14.446499 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.446461 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.454753 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.454935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.454935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.454935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.454935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.455131 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454960 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.455131 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.454998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.455131 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.455074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.455244 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.455133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.556655 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.556655 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.556969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557008 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.557662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.557583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.559300 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.559275 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.559439 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.559418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.570552 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.570520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.570676 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.570584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg\") pod \"e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.716542 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.716510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:14.849718 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.849678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.994359 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.994330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:58:14.996508 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:58:14.996478 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc1191_908a_4651_8499_c9572d41daaf.slice/crio-feee490ecd435337c59d71b2e22f48701f63bb250284e8cd00c1363b50fceb54 WatchSource:0}: Error finding container feee490ecd435337c59d71b2e22f48701f63bb250284e8cd00c1363b50fceb54: Status 404 returned error can't find the container with id feee490ecd435337c59d71b2e22f48701f63bb250284e8cd00c1363b50fceb54 Sep 16 13:58:14.997946 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.997916 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.998822 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.998791 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:14.998921 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.998857 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:14.998921 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:14.998903 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:15.834155 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:15.834113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerStarted","Data":"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4"} Sep 16 13:58:15.834560 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:15.834181 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerStarted","Data":"feee490ecd435337c59d71b2e22f48701f63bb250284e8cd00c1363b50fceb54"} Sep 16 13:58:15.857453 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:15.857406 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podStartSLOduration=1.857392223 podStartE2EDuration="1.857392223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:15.855087919 +0000 UTC m=+914.895681872" watchObservedRunningTime="2026-09-16 13:58:15.857392223 +0000 UTC m=+914.897986174" Sep 16 13:58:15.898704 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:15.898671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:16.717645 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:16.717609 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:16.722841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:16.722814 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:16.761986 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:16.761956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:16.840277 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:16.840242 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:16.841119 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:16.841100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:17.224910 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.224856 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.325392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.325361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.577884 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.577790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.626741 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.626710 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:58:17.626953 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.626926 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" podUID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" containerName="payload-pre-processing" containerID="cri-o://5e6b72532a8d2db62da83f5f6a4654cea6ea849a8422d9d4b40a3ce493002eab" gracePeriod=30 Sep 16 13:58:17.671565 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.671526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.843328 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.843233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.848299 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.848271 2559 generic.go:356] "Generic (PLEG): container finished" podID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" containerID="5e6b72532a8d2db62da83f5f6a4654cea6ea849a8422d9d4b40a3ce493002eab" exitCode=0 Sep 16 13:58:17.848439 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.848353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" event={"ID":"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc","Type":"ContainerDied","Data":"5e6b72532a8d2db62da83f5f6a4654cea6ea849a8422d9d4b40a3ce493002eab"} Sep 16 13:58:17.877055 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.877029 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:58:17.976137 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.976102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:17.991206 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.991176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvd4j\" (UniqueName: \"kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j\") pod \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " Sep 16 13:58:17.991377 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.991313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume\") pod \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\" (UID: \"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc\") " Sep 16 13:58:17.991656 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.991635 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" (UID: "fe7d9db1-49b2-4f29-b3f9-751da67cf8cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:17.993368 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:17.993344 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j" (OuterVolumeSpecName: "kube-api-access-gvd4j") pod "fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" (UID: "fe7d9db1-49b2-4f29-b3f9-751da67cf8cc"). InnerVolumeSpecName "kube-api-access-gvd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:18.076273 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.076240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:18.099296 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.093342 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:18.099296 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.093378 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd4j\" (UniqueName: \"kubernetes.io/projected/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc-kube-api-access-gvd4j\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:18.338354 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.338320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:18.854791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.854753 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" Sep 16 13:58:18.854791 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.854782 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt" event={"ID":"fe7d9db1-49b2-4f29-b3f9-751da67cf8cc","Type":"ContainerDied","Data":"25409fae97a4c0503401455cf5af7969c3b60941ed7135c629e3101795975304"} Sep 16 13:58:18.855357 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.854846 2559 scope.go:117] "RemoveContainer" containerID="5e6b72532a8d2db62da83f5f6a4654cea6ea849a8422d9d4b40a3ce493002eab" Sep 16 13:58:18.879044 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.879014 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:58:18.883794 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.883770 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-617597-94db8499d-lkdbt"] Sep 16 13:58:18.995521 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:18.995487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:19.471475 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:19.471441 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" path="/var/lib/kubelet/pods/fe7d9db1-49b2-4f29-b3f9-751da67cf8cc/volumes" Sep 16 13:58:19.780114 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:19.780029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:20.704929 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:20.704859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:21.485252 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:21.485211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:22.325469 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:22.325404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:22.402515 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:22.402482 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:22.533995 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:22.533962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:22.940494 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:22.940456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:23.037788 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:23.037752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:23.412090 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:23.412057 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:58:23.508524 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:23.508486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:23.608660 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:23.608626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:24.416453 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:24.416413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:31.157159 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:31.157129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:31.853987 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:31.853949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:58:38.431294 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.431246 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" podUID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" containerName="main" containerID="cri-o://fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67" gracePeriod=45 Sep 16 13:58:38.728884 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.728834 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:58:38.802794 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802762 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802837 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802893 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh2gz\" (UniqueName: \"kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802923 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802952 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803024 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.802974 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm\") pod \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\" (UID: \"7f1c36d6-5802-4d43-a7c3-30b060b93dc3\") " Sep 16 13:58:38.803311 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803082 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:38.803311 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803140 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:38.803311 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803160 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home" (OuterVolumeSpecName: "home") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:38.803441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803334 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.803441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803352 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.803441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.803365 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.805219 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.805194 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm" (OuterVolumeSpecName: "dshm") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:38.805288 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.805257 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:38.805322 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.805282 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz" (OuterVolumeSpecName: "kube-api-access-gh2gz") pod "7f1c36d6-5802-4d43-a7c3-30b060b93dc3" (UID: "7f1c36d6-5802-4d43-a7c3-30b060b93dc3"). InnerVolumeSpecName "kube-api-access-gh2gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:38.904684 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.904639 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2gz\" (UniqueName: \"kubernetes.io/projected/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-kube-api-access-gh2gz\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.904684 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.904678 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.904684 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.904689 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f1c36d6-5802-4d43-a7c3-30b060b93dc3-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:38.962341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.962244 2559 generic.go:356] "Generic (PLEG): container finished" podID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" containerID="fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67" exitCode=0 Sep 16 13:58:38.962341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.962286 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" Sep 16 13:58:38.962341 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.962287 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" event={"ID":"7f1c36d6-5802-4d43-a7c3-30b060b93dc3","Type":"ContainerDied","Data":"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67"} Sep 16 13:58:38.962581 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.962352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2" event={"ID":"7f1c36d6-5802-4d43-a7c3-30b060b93dc3","Type":"ContainerDied","Data":"befe4d4e6c3caac5bef9fd2ca75f6a15c9e7b8bd92a64d3c7b98f9ed83e9120c"} Sep 16 13:58:38.962581 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.962376 2559 scope.go:117] "RemoveContainer" containerID="fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67" Sep 16 13:58:38.972310 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.972281 2559 scope.go:117] "RemoveContainer" containerID="fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67" Sep 16 13:58:38.972697 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:58:38.972660 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67\": container with ID starting with fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67 not found: ID does not exist" containerID="fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67" Sep 16 13:58:38.972780 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.972706 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67"} err="failed to get container status \"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67\": rpc error: code = NotFound desc = could not find container \"fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67\": container with ID starting with fde0af3c7d699032df465d27c6026ff66f9aab91866893b2b9c6f81316e6cb67 not found: ID does not exist" Sep 16 13:58:38.986256 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.986213 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:58:38.993992 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:38.993960 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-617597/autores-model-617597-kserve-57d78cb5d7-6hbt2"] Sep 16 13:58:39.471971 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:39.471939 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" path="/var/lib/kubelet/pods/7f1c36d6-5802-4d43-a7c3-30b060b93dc3/volumes" Sep 16 13:58:42.986034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.985997 2559 generic.go:356] "Generic (PLEG): container finished" podID="2bcc1191-908a-4651-8499-c9572d41daaf" containerID="6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4" exitCode=0 Sep 16 13:58:42.986514 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.986073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerDied","Data":"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4"} Sep 16 13:58:42.986514 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.986511 2559 scope.go:117] "RemoveContainer" containerID="6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4" Sep 16 13:58:42.987643 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.987616 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:42.987695 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.987681 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:42.987741 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:42.987713 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:58:43.992699 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:43.992659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerStarted","Data":"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06"} Sep 16 13:58:44.717252 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:44.717220 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:44.718356 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:44.718314 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 13:58:44.718568 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:44.718541 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 13:58:44.737680 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:44.737650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:45.717857 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:45.717814 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 13:58:45.718316 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:45.717903 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 13:58:45.759688 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:45.759649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:46.704398 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:46.704361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:46.717537 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:46.717500 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 13:58:46.717701 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:46.717569 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 13:58:46.840301 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:46.840266 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:47.170958 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.170840 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw"] Sep 16 13:58:47.171486 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171451 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" containerName="main" Sep 16 13:58:47.171486 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171479 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" containerName="main" Sep 16 13:58:47.171697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171532 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" containerName="payload-pre-processing" Sep 16 13:58:47.171697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171541 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" containerName="payload-pre-processing" Sep 16 13:58:47.171697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171644 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1c36d6-5802-4d43-a7c3-30b060b93dc3" containerName="main" Sep 16 13:58:47.171697 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.171656 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe7d9db1-49b2-4f29-b3f9-751da67cf8cc" containerName="payload-pre-processing" Sep 16 13:58:47.174288 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.174269 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:47.176549 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.176524 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 13:58:47.176757 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.176736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:58:47.183279 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.183250 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw"] Sep 16 13:58:47.293109 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.293062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw\" (UID: \"002a52b8-6963-428c-86a5-69748556a7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:47.394389 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.394353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw\" (UID: \"002a52b8-6963-428c-86a5-69748556a7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:47.394996 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.394976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw\" (UID: \"002a52b8-6963-428c-86a5-69748556a7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:47.485589 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.485551 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:47.717241 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.717196 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 13:58:47.717414 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.717258 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 13:58:47.774638 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.774608 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw"] Sep 16 13:58:47.777245 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:58:47.777213 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002a52b8_6963_428c_86a5_69748556a7e4.slice/crio-2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959 WatchSource:0}: Error finding container 2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959: Status 404 returned error can't find the container with id 2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959 Sep 16 13:58:47.837557 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:47.837528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:48.015775 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:48.015740 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" event={"ID":"002a52b8-6963-428c-86a5-69748556a7e4","Type":"ContainerStarted","Data":"2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959"} Sep 16 13:58:48.717493 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:48.717456 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 16 13:58:48.717681 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:48.717515 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 16 13:58:48.897058 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:48.897026 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:49.022743 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:49.022659 2559 generic.go:356] "Generic (PLEG): container finished" podID="002a52b8-6963-428c-86a5-69748556a7e4" containerID="5f2138d1ceee3e1d5f99ad66ccbd5a5a5b98a648dc6c9177a5ed9eb9cdb886cf" exitCode=0 Sep 16 13:58:49.023103 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:49.022744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" event={"ID":"002a52b8-6963-428c-86a5-69748556a7e4","Type":"ContainerDied","Data":"5f2138d1ceee3e1d5f99ad66ccbd5a5a5b98a648dc6c9177a5ed9eb9cdb886cf"} Sep 16 13:58:49.721417 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:49.721383 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:49.722475 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:49.722453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:58:49.845648 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:49.845614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:50.161499 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:50.161472 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:50.222036 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:50.222001 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca\") pod \"002a52b8-6963-428c-86a5-69748556a7e4\" (UID: \"002a52b8-6963-428c-86a5-69748556a7e4\") " Sep 16 13:58:50.222386 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:50.222363 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "002a52b8-6963-428c-86a5-69748556a7e4" (UID: "002a52b8-6963-428c-86a5-69748556a7e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:50.323507 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:50.323403 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002a52b8-6963-428c-86a5-69748556a7e4-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:58:50.743587 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:50.743557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:51.035424 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:51.035341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" Sep 16 13:58:51.035573 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:51.035340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw" event={"ID":"002a52b8-6963-428c-86a5-69748556a7e4","Type":"ContainerDied","Data":"2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959"} Sep 16 13:58:51.035573 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:51.035473 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e3bafd6a36265375241e907f2c2bbd59b90f1670484f61edf87e2d096c00959" Sep 16 13:58:51.894541 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:51.894510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:52.919290 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:52.919253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:58.668443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:58.668400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:59.695214 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:59.695172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:59.873201 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:59.873168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:59.967772 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:58:59.967674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:00.155494 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.155449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:00.191768 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.191735 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw"] Sep 16 13:59:00.216373 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.216338 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1852eb66-dx4bw"] Sep 16 13:59:00.243974 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.243889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:00.353277 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.353246 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:59:00.353470 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.353434 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" containerID="cri-o://1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06" gracePeriod=30 Sep 16 13:59:00.393615 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.393577 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:00.661166 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:00.661070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:01.120499 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:01.120465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:01.238904 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:01.238839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:01.472193 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:01.472156 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002a52b8-6963-428c-86a5-69748556a7e4" path="/var/lib/kubelet/pods/002a52b8-6963-428c-86a5-69748556a7e4/volumes" Sep 16 13:59:02.158396 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:02.158360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:02.696497 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:02.696465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:03.097751 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:03.097660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:03.787098 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:03.787069 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:04.077255 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.077121 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:59:04.077433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.077410 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" containerID="cri-o://39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b" gracePeriod=30 Sep 16 13:59:04.113190 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.113138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:04.416452 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.416323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:04.614154 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.614094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:04.724670 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.724633 2559 patch_prober.go:28] interesting pod/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:59:04.724825 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.724702 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:59:04.947071 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:04.947033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:05.156711 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.156624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:05.454470 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.454429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:05.616402 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.616374 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:59:05.779091 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779004 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779091 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779042 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779091 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779182 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779390 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779366 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:05.779662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779443 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779468 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket\") pod \"2bcc1191-908a-4651-8499-c9572d41daaf\" (UID: \"2bcc1191-908a-4651-8499-c9572d41daaf\") " Sep 16 13:59:05.779662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:05.779662 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779552 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:59:05.779903 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779763 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:05.779903 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779776 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:05.780017 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.779998 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.780077 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.780017 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.780077 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.780033 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.780077 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.780049 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc1191-908a-4651-8499-c9572d41daaf-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.780077 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.780061 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.781483 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.781450 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg" (OuterVolumeSpecName: "kube-api-access-fxbdg") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "kube-api-access-fxbdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:05.781610 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.781548 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:05.781679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.781605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:05.781679 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.781619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bcc1191-908a-4651-8499-c9572d41daaf" (UID: "2bcc1191-908a-4651-8499-c9572d41daaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:59:05.881434 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.881398 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc1191-908a-4651-8499-c9572d41daaf-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.881434 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.881429 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc1191-908a-4651-8499-c9572d41daaf-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.881434 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.881438 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbdg\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-kube-api-access-fxbdg\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:05.881672 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:05.881448 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc1191-908a-4651-8499-c9572d41daaf-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:06.114457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.114358 2559 generic.go:356] "Generic (PLEG): container finished" podID="2bcc1191-908a-4651-8499-c9572d41daaf" containerID="1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06" exitCode=0 Sep 16 13:59:06.114457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.114399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerDied","Data":"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06"} Sep 16 13:59:06.114457 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.114409 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" Sep 16 13:59:06.115014 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.114462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt" event={"ID":"2bcc1191-908a-4651-8499-c9572d41daaf","Type":"ContainerDied","Data":"feee490ecd435337c59d71b2e22f48701f63bb250284e8cd00c1363b50fceb54"} Sep 16 13:59:06.115014 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.114486 2559 scope.go:117] "RemoveContainer" containerID="1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06" Sep 16 13:59:06.125826 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.125805 2559 scope.go:117] "RemoveContainer" containerID="6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4" Sep 16 13:59:06.134124 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.134098 2559 scope.go:117] "RemoveContainer" containerID="1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06" Sep 16 13:59:06.134387 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:59:06.134369 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06\": container with ID starting with 1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06 not found: ID does not exist" containerID="1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06" Sep 16 13:59:06.134462 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.134396 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06"} err="failed to get container status \"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06\": rpc error: code = NotFound desc = could not find container \"1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06\": container with ID starting with 1c70312633a2dd99a3d5a5ec1281d2743e072fa36a66a7627151596ec1c86d06 not found: ID does not exist" Sep 16 13:59:06.134462 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.134412 2559 scope.go:117] "RemoveContainer" containerID="6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4" Sep 16 13:59:06.134653 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:59:06.134635 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4\": container with ID starting with 6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4 not found: ID does not exist" containerID="6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4" Sep 16 13:59:06.134694 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.134658 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4"} err="failed to get container status \"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4\": rpc error: code = NotFound desc = could not find container \"6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4\": container with ID starting with 6dd2e3d256a630ffcf352d722cde5b50011cf28babb0216948053aa0f6034aa4 not found: ID does not exist" Sep 16 13:59:06.140838 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.140808 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:59:06.145630 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.145607 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1852eb66-openshift-default-65ddbb4d88-jrklt"] Sep 16 13:59:06.858380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:06.858342 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:59:07.472103 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:07.472067 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" path="/var/lib/kubelet/pods/2bcc1191-908a-4651-8499-c9572d41daaf/volumes" Sep 16 13:59:07.771151 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:07.771047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:08.320725 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:08.320679 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:08.416582 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:08.416547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:08.669380 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:08.669289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:08.881903 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:08.881848 2559 patch_prober.go:28] interesting pod/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:59:08.882086 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:08.881922 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:59:09.336841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.336815 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:59:09.417034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.416953 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417003 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417278 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417278 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417085 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417278 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417270 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:09.417443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417377 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data" (OuterVolumeSpecName: "istio-data") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:09.417443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417395 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:09.417608 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417530 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417608 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417559 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f67c\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417608 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417588 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket\") pod \"a51aa693-e4d3-40ad-80e0-bb92455fa620\" (UID: \"a51aa693-e4d3-40ad-80e0-bb92455fa620\") " Sep 16 13:59:09.417749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417609 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:59:09.417953 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417930 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:09.418045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.417982 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.418045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.418004 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.418045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.418018 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.418045 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.418031 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51aa693-e4d3-40ad-80e0-bb92455fa620-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.419626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.419600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c" (OuterVolumeSpecName: "kube-api-access-2f67c") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "kube-api-access-2f67c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:09.419744 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.419726 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token" (OuterVolumeSpecName: "istio-token") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:09.419809 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.419737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:09.420029 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.420012 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a51aa693-e4d3-40ad-80e0-bb92455fa620" (UID: "a51aa693-e4d3-40ad-80e0-bb92455fa620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:59:09.519292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.519261 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.519292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.519290 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.519292 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.519301 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51aa693-e4d3-40ad-80e0-bb92455fa620-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.519530 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.519310 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f67c\" (UniqueName: \"kubernetes.io/projected/a51aa693-e4d3-40ad-80e0-bb92455fa620-kube-api-access-2f67c\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.519530 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.519320 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51aa693-e4d3-40ad-80e0-bb92455fa620-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 13:59:09.860632 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:09.860594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.138584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.138490 2559 generic.go:356] "Generic (PLEG): container finished" podID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerID="39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b" exitCode=0 Sep 16 13:59:10.138584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.138550 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" Sep 16 13:59:10.138584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.138566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerDied","Data":"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b"} Sep 16 13:59:10.138821 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.138620 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6" event={"ID":"a51aa693-e4d3-40ad-80e0-bb92455fa620","Type":"ContainerDied","Data":"c6d0778a9cea991e798f6333b81b373fd12958e952e84c7de7ae002ec0e5e398"} Sep 16 13:59:10.138821 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.138643 2559 scope.go:117] "RemoveContainer" containerID="39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b" Sep 16 13:59:10.147402 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.147384 2559 scope.go:117] "RemoveContainer" containerID="4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9" Sep 16 13:59:10.155417 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.155404 2559 scope.go:117] "RemoveContainer" containerID="39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b" Sep 16 13:59:10.155711 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:59:10.155687 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b\": container with ID starting with 39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b not found: ID does not exist" containerID="39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b" Sep 16 13:59:10.155766 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.155727 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b"} err="failed to get container status \"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b\": rpc error: code = NotFound desc = could not find container \"39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b\": container with ID starting with 39c40abd72b2ad1a899464f46678be809cdfa0bd3b4cd26bdb088d24d4c4383b not found: ID does not exist" Sep 16 13:59:10.155766 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.155745 2559 scope.go:117] "RemoveContainer" containerID="4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9" Sep 16 13:59:10.156036 ip-10-0-139-34 kubenswrapper[2559]: E0916 13:59:10.156017 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9\": container with ID starting with 4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9 not found: ID does not exist" containerID="4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9" Sep 16 13:59:10.156102 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.156040 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9"} err="failed to get container status \"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9\": rpc error: code = NotFound desc = could not find container \"4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9\": container with ID starting with 4b09d037e21f40cf49bff82139a8dfcfcf3cf30f7f3cf1d5fd585dbb37e711b9 not found: ID does not exist" Sep 16 13:59:10.164047 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.164024 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:59:10.169350 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.169327 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-617597-openshift-default-5cdd69f55b-xqcc6"] Sep 16 13:59:10.235613 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.235581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.334083 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.334050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.446251 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.446215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.667056 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.667016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.970935 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:10.970905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:11.112022 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:11.111993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:11.472888 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:11.472847 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" path="/var/lib/kubelet/pods/a51aa693-e4d3-40ad-80e0-bb92455fa620/volumes" Sep 16 13:59:12.201099 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:12.201067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.033720 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.033680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.130989 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.130947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.483153 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.483117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.595607 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.595548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.691358 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.691319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:13.809020 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.808940 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 13:59:13.809351 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809337 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002a52b8-6963-428c-86a5-69748556a7e4" containerName="revoke-keys" Sep 16 13:59:13.809392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809353 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="002a52b8-6963-428c-86a5-69748556a7e4" containerName="revoke-keys" Sep 16 13:59:13.809392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809371 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809376 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809385 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809392 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809392 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809402 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809408 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809415 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809420 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809489 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809497 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcc1191-908a-4651-8499-c9572d41daaf" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809505 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809512 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51aa693-e4d3-40ad-80e0-bb92455fa620" containerName="istio-proxy" Sep 16 13:59:13.809544 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.809518 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="002a52b8-6963-428c-86a5-69748556a7e4" containerName="revoke-keys" Sep 16 13:59:13.813858 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.813843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:13.816244 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.816217 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-733e5b\"" Sep 16 13:59:13.816378 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.816256 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-733e5b-dockercfg-hwstz\"" Sep 16 13:59:13.821892 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.821841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 13:59:13.886908 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.886862 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 13:59:13.890158 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.890140 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:13.900808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.900779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 13:59:13.959185 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.959155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:13.959360 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:13.959220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mkv\" (UniqueName: \"kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.060498 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.060400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.060498 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.060459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlql\" (UniqueName: \"kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.060498 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.060494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mkv\" (UniqueName: \"kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.060797 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.060577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.061217 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.061193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.077496 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.077465 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mkv\" (UniqueName: \"kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv\") pod \"payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.127183 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.127134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:14.162128 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.162090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlql\" (UniqueName: \"kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.162288 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.162144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.162765 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.162747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.171758 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.171732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlql\" (UniqueName: \"kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql\") pod \"payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.202197 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.202165 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:14.399621 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.399519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 13:59:14.402522 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:59:14.402491 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caab58c_868a_4aff_b1da_36363ae721a4.slice/crio-cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c WatchSource:0}: Error finding container cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c: Status 404 returned error can't find the container with id cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c Sep 16 13:59:14.479179 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:14.479143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 13:59:14.482374 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:59:14.482351 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ca81e4_7908_49ab_a9e6_38da7f20bd9b.slice/crio-5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1 WatchSource:0}: Error finding container 5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1: Status 404 returned error can't find the container with id 5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1 Sep 16 13:59:15.166506 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.166446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" event={"ID":"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b","Type":"ContainerStarted","Data":"d11fe8d98a11c551655fcf644fe0509811417bfb727a789e187861b2bb154a3f"} Sep 16 13:59:15.166979 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.166513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" event={"ID":"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b","Type":"ContainerStarted","Data":"5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1"} Sep 16 13:59:15.166979 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.166545 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:15.168479 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.168455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" event={"ID":"4caab58c-868a-4aff-b1da-36363ae721a4","Type":"ContainerStarted","Data":"8f0bd64037a9efcabee10cec89c7e2e2fc49eeb2d4746c793043afe6713f7a47"} Sep 16 13:59:15.168600 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.168496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" event={"ID":"4caab58c-868a-4aff-b1da-36363ae721a4","Type":"ContainerStarted","Data":"cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c"} Sep 16 13:59:15.168654 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.168627 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:15.186623 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.186567 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" podStartSLOduration=2.186548737 podStartE2EDuration="2.186548737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:15.185651445 +0000 UTC m=+974.226245403" watchObservedRunningTime="2026-09-16 13:59:15.186548737 +0000 UTC m=+974.227142689" Sep 16 13:59:15.207995 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:15.207942 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" podStartSLOduration=2.207923346 podStartE2EDuration="2.207923346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:15.20556917 +0000 UTC m=+974.246163124" watchObservedRunningTime="2026-09-16 13:59:15.207923346 +0000 UTC m=+974.248517350" Sep 16 13:59:20.460853 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.460814 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 13:59:20.464590 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.464569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.467035 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.467012 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"premium-w4-19203d-kserve-self-signed-certs\"" Sep 16 13:59:20.478326 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.478299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 13:59:20.517062 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5z8\" (UniqueName: \"kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.517269 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517108 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.517269 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.517269 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.517466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.517466 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.517385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618260 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5z8\" (UniqueName: \"kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618626 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618724 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.618808 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.618803 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.620560 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.620539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.620803 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.620783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.632207 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.632179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5z8\" (UniqueName: \"kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8\") pod \"premium-w4-19203d-kserve-997cc74d9-pn62s\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:20.752100 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.751999 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:20.773996 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:20.773957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:21.049665 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:21.049638 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 13:59:21.052054 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:59:21.052027 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a82cfad_f3c8_4241_bfb4_137d4d8e5218.slice/crio-71c7d4d757c9501948344a08d7d62dd7ac8efb32fa9da7aacfcbef6d803fff28 WatchSource:0}: Error finding container 71c7d4d757c9501948344a08d7d62dd7ac8efb32fa9da7aacfcbef6d803fff28: Status 404 returned error can't find the container with id 71c7d4d757c9501948344a08d7d62dd7ac8efb32fa9da7aacfcbef6d803fff28 Sep 16 13:59:21.200712 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:21.200673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" event={"ID":"3a82cfad-f3c8-4241-bfb4-137d4d8e5218","Type":"ContainerStarted","Data":"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586"} Sep 16 13:59:21.200918 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:21.200728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" event={"ID":"3a82cfad-f3c8-4241-bfb4-137d4d8e5218","Type":"ContainerStarted","Data":"71c7d4d757c9501948344a08d7d62dd7ac8efb32fa9da7aacfcbef6d803fff28"} Sep 16 13:59:21.227433 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:21.227382 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" podStartSLOduration=1.227368459 podStartE2EDuration="1.227368459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:21.224715745 +0000 UTC m=+980.265309698" watchObservedRunningTime="2026-09-16 13:59:21.227368459 +0000 UTC m=+980.267962412" Sep 16 13:59:26.175576 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:26.175496 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 13:59:26.176143 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:26.175650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 13:59:30.774642 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:30.774592 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:30.774642 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:30.774646 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:30.786749 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:30.786724 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:31.146584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.146508 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 13:59:31.150122 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.150106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.152584 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.152562 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"premium-w0-035c64-kserve-self-signed-certs\"" Sep 16 13:59:31.163129 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.163103 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 13:59:31.181973 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.181943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:31.210236 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.210338 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.210338 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcml\" (UniqueName: \"kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.210338 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.210454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.210454 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.210411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.268426 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.268396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 13:59:31.318504 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.318688 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcml\" (UniqueName: \"kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.318688 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.318824 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.318824 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.318824 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.318801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.319355 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.319327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.319554 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.319531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.320029 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.320004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.327741 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.327701 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.328231 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.328209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.334543 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.334514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcml\" (UniqueName: \"kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml\") pod \"premium-w0-035c64-kserve-65f4d4f48f-gm7xc\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.363162 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.363132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:59:31.459427 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.459393 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:31.744963 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:31.744936 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 13:59:31.746778 ip-10-0-139-34 kubenswrapper[2559]: W0916 13:59:31.746748 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e42127_44de_442a_bfc9_47b4135510ef.slice/crio-db0cac2b1ffafb3cc6e12628f0c20d239431531a8f26557c7492b1b0666820a9 WatchSource:0}: Error finding container db0cac2b1ffafb3cc6e12628f0c20d239431531a8f26557c7492b1b0666820a9: Status 404 returned error can't find the container with id db0cac2b1ffafb3cc6e12628f0c20d239431531a8f26557c7492b1b0666820a9 Sep 16 13:59:32.262920 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:32.262875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" event={"ID":"33e42127-44de-442a-bfc9-47b4135510ef","Type":"ContainerStarted","Data":"cd9c5cfb94b125040a77092357ee8f9ef9324ee8217c0cb14e0a23487f333af5"} Sep 16 13:59:32.263373 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:32.262929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" event={"ID":"33e42127-44de-442a-bfc9-47b4135510ef","Type":"ContainerStarted","Data":"db0cac2b1ffafb3cc6e12628f0c20d239431531a8f26557c7492b1b0666820a9"} Sep 16 13:59:32.284109 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:32.284041 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" podStartSLOduration=1.28400777 podStartE2EDuration="1.28400777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:32.280951565 +0000 UTC m=+991.321545519" watchObservedRunningTime="2026-09-16 13:59:32.28400777 +0000 UTC m=+991.324601726" Sep 16 13:59:35.529308 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:35.529273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:36.493343 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:36.493291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:36.926929 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:36.926782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:37.394784 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:37.394717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:38.209441 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:38.209397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:38.360207 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:38.360170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:39.100563 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:39.100529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:40.089034 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:40.088998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:40.717443 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:40.717401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:41.459841 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:41.459801 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:41.460332 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:41.460000 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:41.473640 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:41.473612 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:42.325442 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:42.325410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 13:59:43.789919 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:43.789882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:43.899605 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:43.899560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:43.994894 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:43.994837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:50.964374 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:50.964337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 13:59:58.910577 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:58.910543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:59.007127 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:59.007090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:59.648646 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:59.648599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:59.893758 ip-10-0-139-34 kubenswrapper[2559]: I0916 13:59:59.893724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:00.153235 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.153203 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r"] Sep 16 14:00:00.156687 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.156658 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8"] Sep 16 14:00:00.156838 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.156776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" Sep 16 14:00:00.160079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.160057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" Sep 16 14:00:00.160618 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.160595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:00:00.168805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.168783 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" Sep 16 14:00:00.169932 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.169812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8"] Sep 16 14:00:00.173370 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.173346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r"] Sep 16 14:00:00.178699 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.178677 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" Sep 16 14:00:00.190596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.190566 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5"] Sep 16 14:00:00.195045 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.195023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" Sep 16 14:00:00.209241 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.209207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5"] Sep 16 14:00:00.230892 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.230832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" Sep 16 14:00:00.472609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.472580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r"] Sep 16 14:00:00.474584 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.474560 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8"] Sep 16 14:00:00.474689 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:00.474667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8895fcb_d369_4064_aba7_0b20dd2d4e19.slice/crio-d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4 WatchSource:0}: Error finding container d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4: Status 404 returned error can't find the container with id d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4 Sep 16 14:00:00.475307 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:00.475289 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79613e6c_8a7f_42f1_b3ac_19eb77c8189c.slice/crio-6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7 WatchSource:0}: Error finding container 6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7: Status 404 returned error can't find the container with id 6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7 Sep 16 14:00:00.530439 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:00.530406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5"] Sep 16 14:00:00.532257 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:00.532228 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d84384_58ce_46ab_8b0a_ba00c7348bb5.slice/crio-8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390 WatchSource:0}: Error finding container 8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390: Status 404 returned error can't find the container with id 8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390 Sep 16 14:00:01.416923 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.416853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" event={"ID":"f5d84384-58ce-46ab-8b0a-ba00c7348bb5","Type":"ContainerStarted","Data":"4eb2b936404aa959bf4f0bda158a983990274be77f32ee911325e3b2b1e60c92"} Sep 16 14:00:01.417421 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.416954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" event={"ID":"f5d84384-58ce-46ab-8b0a-ba00c7348bb5","Type":"ContainerStarted","Data":"8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390"} Sep 16 14:00:01.418443 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.418415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" event={"ID":"79613e6c-8a7f-42f1-b3ac-19eb77c8189c","Type":"ContainerStarted","Data":"25c78db7dec2eb8f6b0e2041bfac993c422ec3d298af332b3fa027c70d12f317"} Sep 16 14:00:01.418565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.418462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" event={"ID":"79613e6c-8a7f-42f1-b3ac-19eb77c8189c","Type":"ContainerStarted","Data":"6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7"} Sep 16 14:00:01.420017 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.419993 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" event={"ID":"d8895fcb-d369-4064-aba7-0b20dd2d4e19","Type":"ContainerStarted","Data":"fecf7d4cfbc364263832c6b15071d0c508663b8c3452e43cc8bf0e176200b5cc"} Sep 16 14:00:01.420129 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.420039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" event={"ID":"d8895fcb-d369-4064-aba7-0b20dd2d4e19","Type":"ContainerStarted","Data":"d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4"} Sep 16 14:00:01.436669 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.436601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" podStartSLOduration=1.436578948 podStartE2EDuration="1.436578948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.433335998 +0000 UTC m=+1020.473929954" watchObservedRunningTime="2026-09-16 14:00:01.436578948 +0000 UTC m=+1020.477172903" Sep 16 14:00:01.450372 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.450312 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" podStartSLOduration=1.450291583 podStartE2EDuration="1.450291583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.448700629 +0000 UTC m=+1020.489294585" watchObservedRunningTime="2026-09-16 14:00:01.450291583 +0000 UTC m=+1020.490885539" Sep 16 14:00:01.468172 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:01.468113 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" podStartSLOduration=1.468092787 podStartE2EDuration="1.468092787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.465326775 +0000 UTC m=+1020.505920731" watchObservedRunningTime="2026-09-16 14:00:01.468092787 +0000 UTC m=+1020.508686747" Sep 16 14:00:02.427227 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.427194 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5d84384-58ce-46ab-8b0a-ba00c7348bb5" containerID="4eb2b936404aa959bf4f0bda158a983990274be77f32ee911325e3b2b1e60c92" exitCode=0 Sep 16 14:00:02.427673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.427284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" event={"ID":"f5d84384-58ce-46ab-8b0a-ba00c7348bb5","Type":"ContainerDied","Data":"4eb2b936404aa959bf4f0bda158a983990274be77f32ee911325e3b2b1e60c92"} Sep 16 14:00:02.428766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.428743 2559 generic.go:356] "Generic (PLEG): container finished" podID="79613e6c-8a7f-42f1-b3ac-19eb77c8189c" containerID="25c78db7dec2eb8f6b0e2041bfac993c422ec3d298af332b3fa027c70d12f317" exitCode=0 Sep 16 14:00:02.428889 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.428810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" event={"ID":"79613e6c-8a7f-42f1-b3ac-19eb77c8189c","Type":"ContainerDied","Data":"25c78db7dec2eb8f6b0e2041bfac993c422ec3d298af332b3fa027c70d12f317"} Sep 16 14:00:02.430220 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.430195 2559 generic.go:356] "Generic (PLEG): container finished" podID="d8895fcb-d369-4064-aba7-0b20dd2d4e19" containerID="fecf7d4cfbc364263832c6b15071d0c508663b8c3452e43cc8bf0e176200b5cc" exitCode=0 Sep 16 14:00:02.430324 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.430272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" event={"ID":"d8895fcb-d369-4064-aba7-0b20dd2d4e19","Type":"ContainerDied","Data":"fecf7d4cfbc364263832c6b15071d0c508663b8c3452e43cc8bf0e176200b5cc"} Sep 16 14:00:02.526249 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.526213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:02.669524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:02.669491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:03.591637 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:03.591609 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" Sep 16 14:00:03.642893 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:03.642849 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" Sep 16 14:00:03.645864 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:03.645842 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" Sep 16 14:00:04.442906 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.442861 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" Sep 16 14:00:04.443079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.442860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5" event={"ID":"f5d84384-58ce-46ab-8b0a-ba00c7348bb5","Type":"ContainerDied","Data":"8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390"} Sep 16 14:00:04.443079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.442999 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b68af810b1b7d131fdd70706f7ab777fe36e0db0244bd00fbd0c196ab6d7390" Sep 16 14:00:04.444221 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.444192 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" Sep 16 14:00:04.444351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.444197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8" event={"ID":"79613e6c-8a7f-42f1-b3ac-19eb77c8189c","Type":"ContainerDied","Data":"6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7"} Sep 16 14:00:04.444351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.444333 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de2aba36dcb5ae724322ecdf72ed423a38070635eb2b46f1497d2c4dc1e40f7" Sep 16 14:00:04.445504 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.445485 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" Sep 16 14:00:04.445609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.445483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r" event={"ID":"d8895fcb-d369-4064-aba7-0b20dd2d4e19","Type":"ContainerDied","Data":"d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4"} Sep 16 14:00:04.445609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:04.445606 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e92ff118e9bd7a04426e8284b5a96587380f76b301c220d451cbaacb5144f4" Sep 16 14:00:06.215796 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:06.215758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:06.328348 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:06.328313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:06.421444 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:06.421413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:09.678939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:09.678863 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:00:15.172651 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.172590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:15.274030 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.273993 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz"] Sep 16 14:00:15.274437 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274423 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79613e6c-8a7f-42f1-b3ac-19eb77c8189c" containerName="cleanup" Sep 16 14:00:15.274483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274439 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="79613e6c-8a7f-42f1-b3ac-19eb77c8189c" containerName="cleanup" Sep 16 14:00:15.274483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274449 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8895fcb-d369-4064-aba7-0b20dd2d4e19" containerName="cleanup" Sep 16 14:00:15.274483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274456 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8895fcb-d369-4064-aba7-0b20dd2d4e19" containerName="cleanup" Sep 16 14:00:15.274483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274473 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d84384-58ce-46ab-8b0a-ba00c7348bb5" containerName="cleanup" Sep 16 14:00:15.274483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274482 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d84384-58ce-46ab-8b0a-ba00c7348bb5" containerName="cleanup" Sep 16 14:00:15.274650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274557 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8895fcb-d369-4064-aba7-0b20dd2d4e19" containerName="cleanup" Sep 16 14:00:15.274650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274572 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d84384-58ce-46ab-8b0a-ba00c7348bb5" containerName="cleanup" Sep 16 14:00:15.274650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.274579 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="79613e6c-8a7f-42f1-b3ac-19eb77c8189c" containerName="cleanup" Sep 16 14:00:15.277552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.277533 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:15.282086 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.282059 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:00:15.282221 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.282103 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:00:15.286671 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.286643 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz"] Sep 16 14:00:15.348337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.348299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz\" (UID: \"c1074939-c496-4549-b730-b42efe4fb8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:15.449830 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.449795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz\" (UID: \"c1074939-c496-4549-b730-b42efe4fb8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:15.450519 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.450488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz\" (UID: \"c1074939-c496-4549-b730-b42efe4fb8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:15.589554 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.589513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:15.865765 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:15.865736 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz"] Sep 16 14:00:15.868036 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:15.868006 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1074939_c496_4549_b730_b42efe4fb8a9.slice/crio-0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22 WatchSource:0}: Error finding container 0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22: Status 404 returned error can't find the container with id 0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22 Sep 16 14:00:16.240190 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.240123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:16.512448 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.512412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" event={"ID":"c1074939-c496-4549-b730-b42efe4fb8a9","Type":"ContainerStarted","Data":"47d4d039075654e51ea1439efe060d29a6318e29b2fc89377807ce2cd0ec3c49"} Sep 16 14:00:16.512627 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.512466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" event={"ID":"c1074939-c496-4549-b730-b42efe4fb8a9","Type":"ContainerStarted","Data":"0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22"} Sep 16 14:00:16.532008 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.531953 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" podStartSLOduration=1.5319382670000001 podStartE2EDuration="1.531938267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:16.530177945 +0000 UTC m=+1035.570771899" watchObservedRunningTime="2026-09-16 14:00:16.531938267 +0000 UTC m=+1035.572532222" Sep 16 14:00:16.887677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.887584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:16.996337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:16.996303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:17.518797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:17.518763 2559 generic.go:356] "Generic (PLEG): container finished" podID="c1074939-c496-4549-b730-b42efe4fb8a9" containerID="47d4d039075654e51ea1439efe060d29a6318e29b2fc89377807ce2cd0ec3c49" exitCode=0 Sep 16 14:00:17.519267 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:17.518852 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" event={"ID":"c1074939-c496-4549-b730-b42efe4fb8a9","Type":"ContainerDied","Data":"47d4d039075654e51ea1439efe060d29a6318e29b2fc89377807ce2cd0ec3c49"} Sep 16 14:00:17.694975 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:17.694936 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:17.936225 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:17.936135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:18.667435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:18.667407 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:18.784102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:18.784011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca\") pod \"c1074939-c496-4549-b730-b42efe4fb8a9\" (UID: \"c1074939-c496-4549-b730-b42efe4fb8a9\") " Sep 16 14:00:18.784392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:18.784369 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1074939-c496-4549-b730-b42efe4fb8a9" (UID: "c1074939-c496-4549-b730-b42efe4fb8a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:00:18.885218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:18.885185 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1074939-c496-4549-b730-b42efe4fb8a9-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:00:18.951752 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:18.951692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:19.530675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:19.530641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" Sep 16 14:00:19.530850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:19.530679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz" event={"ID":"c1074939-c496-4549-b730-b42efe4fb8a9","Type":"ContainerDied","Data":"0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22"} Sep 16 14:00:19.530850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:19.530726 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6a95208d9b0c9ce2c178426e4ad2a3ed5743464816ac18454bf4d836582e22" Sep 16 14:00:19.826907 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:19.826794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:20.153224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.153147 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:00:20.153552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.153539 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1074939-c496-4549-b730-b42efe4fb8a9" containerName="revoke-keys" Sep 16 14:00:20.153624 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.153553 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1074939-c496-4549-b730-b42efe4fb8a9" containerName="revoke-keys" Sep 16 14:00:20.153658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.153649 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1074939-c496-4549-b730-b42efe4fb8a9" containerName="revoke-keys" Sep 16 14:00:20.156827 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.156801 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.159098 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.159076 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"e2e-conflict-a-b845ebcf-kserve-self-signed-certs\"" Sep 16 14:00:20.166378 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.166353 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:00:20.188853 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.188820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:20.299442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.299626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6zm\" (UniqueName: \"kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.299626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.299626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.299746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.299746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.299659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.401761 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.401761 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6zm\" (UniqueName: \"kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.401996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402300 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.402211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402359 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.402299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.402359 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.402347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.404230 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.404174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.404319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.404240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.411364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.411340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6zm\" (UniqueName: \"kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm\") pod \"e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.467569 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.467529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:20.740332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:20.740298 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:00:20.741934 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:20.741901 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8101a30e_0041_48f5_816c_709de955412a.slice/crio-c2306a3a03f0801b4dcac2cd8b1eb03d4b0fb4dc132cabf9bfb59d9b9c85c4cc WatchSource:0}: Error finding container c2306a3a03f0801b4dcac2cd8b1eb03d4b0fb4dc132cabf9bfb59d9b9c85c4cc: Status 404 returned error can't find the container with id c2306a3a03f0801b4dcac2cd8b1eb03d4b0fb4dc132cabf9bfb59d9b9c85c4cc Sep 16 14:00:21.363484 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:21.363423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:21.544435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:21.544397 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" event={"ID":"8101a30e-0041-48f5-816c-709de955412a","Type":"ContainerStarted","Data":"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c"} Sep 16 14:00:21.544435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:21.544439 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" event={"ID":"8101a30e-0041-48f5-816c-709de955412a","Type":"ContainerStarted","Data":"c2306a3a03f0801b4dcac2cd8b1eb03d4b0fb4dc132cabf9bfb59d9b9c85c4cc"} Sep 16 14:00:21.565342 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:21.565280 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" podStartSLOduration=1.565262518 podStartE2EDuration="1.565262518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:21.56427628 +0000 UTC m=+1040.604870236" watchObservedRunningTime="2026-09-16 14:00:21.565262518 +0000 UTC m=+1040.605856523" Sep 16 14:00:24.553754 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:24.553718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:24.655534 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:24.655497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:24.762256 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:24.762219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:24.861672 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:24.861579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:25.477931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:25.477894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:25.572002 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:25.571969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:25.671934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:25.671863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:26.255088 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.255052 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5"] Sep 16 14:00:26.260116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.260080 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-733e5b-29826120-6kdr5"] Sep 16 14:00:26.296586 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.296547 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 14:00:26.296882 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.296842 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" podUID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" containerName="payload-processing" containerID="cri-o://d11fe8d98a11c551655fcf644fe0509811417bfb727a789e187861b2bb154a3f" gracePeriod=30 Sep 16 14:00:26.305405 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.305376 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 14:00:26.305639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.305578 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" podUID="4caab58c-868a-4aff-b1da-36363ae721a4" containerName="payload-pre-processing" containerID="cri-o://8f0bd64037a9efcabee10cec89c7e2e2fc49eeb2d4746c793043afe6713f7a47" gracePeriod=30 Sep 16 14:00:26.544405 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.544370 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz"] Sep 16 14:00:26.547676 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.547645 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-733e5b-jwcwz"] Sep 16 14:00:26.574040 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.574013 2559 generic.go:356] "Generic (PLEG): container finished" podID="4caab58c-868a-4aff-b1da-36363ae721a4" containerID="8f0bd64037a9efcabee10cec89c7e2e2fc49eeb2d4746c793043afe6713f7a47" exitCode=0 Sep 16 14:00:26.574379 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.574088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" event={"ID":"4caab58c-868a-4aff-b1da-36363ae721a4","Type":"ContainerDied","Data":"8f0bd64037a9efcabee10cec89c7e2e2fc49eeb2d4746c793043afe6713f7a47"} Sep 16 14:00:26.574379 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.574148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" event={"ID":"4caab58c-868a-4aff-b1da-36363ae721a4","Type":"ContainerDied","Data":"cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c"} Sep 16 14:00:26.574379 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.574178 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda5322d0776fc059a8cc7ea608362f4c9f917b694118aeffffeccaaa4b1f66c" Sep 16 14:00:26.575443 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.575422 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" containerID="d11fe8d98a11c551655fcf644fe0509811417bfb727a789e187861b2bb154a3f" exitCode=0 Sep 16 14:00:26.575527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.575489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" event={"ID":"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b","Type":"ContainerDied","Data":"d11fe8d98a11c551655fcf644fe0509811417bfb727a789e187861b2bb154a3f"} Sep 16 14:00:26.575576 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.575527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" event={"ID":"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b","Type":"ContainerDied","Data":"5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1"} Sep 16 14:00:26.575576 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.575549 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d365c83b5d8addf3324266fbbf6f1a72568db015bc1690cc1ae962f6590a0f1" Sep 16 14:00:26.587452 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.587432 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 14:00:26.591154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.591132 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 14:00:26.614986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.614945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:26.662018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.661987 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdlql\" (UniqueName: \"kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql\") pod \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " Sep 16 14:00:26.662194 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.662046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume\") pod \"4caab58c-868a-4aff-b1da-36363ae721a4\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " Sep 16 14:00:26.662194 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.662121 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5mkv\" (UniqueName: \"kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv\") pod \"4caab58c-868a-4aff-b1da-36363ae721a4\" (UID: \"4caab58c-868a-4aff-b1da-36363ae721a4\") " Sep 16 14:00:26.662194 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.662163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume\") pod \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\" (UID: \"e2ca81e4-7908-49ab-a9e6-38da7f20bd9b\") " Sep 16 14:00:26.662557 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.662520 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4caab58c-868a-4aff-b1da-36363ae721a4" (UID: "4caab58c-868a-4aff-b1da-36363ae721a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:00:26.662557 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.662550 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" (UID: "e2ca81e4-7908-49ab-a9e6-38da7f20bd9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:00:26.664373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.664345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql" (OuterVolumeSpecName: "kube-api-access-kdlql") pod "e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" (UID: "e2ca81e4-7908-49ab-a9e6-38da7f20bd9b"). InnerVolumeSpecName "kube-api-access-kdlql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:26.664479 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.664376 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv" (OuterVolumeSpecName: "kube-api-access-f5mkv") pod "4caab58c-868a-4aff-b1da-36363ae721a4" (UID: "4caab58c-868a-4aff-b1da-36363ae721a4"). InnerVolumeSpecName "kube-api-access-f5mkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:26.683054 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.683028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:26.763933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.763834 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.763933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.763864 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlql\" (UniqueName: \"kubernetes.io/projected/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b-kube-api-access-kdlql\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.763933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.763889 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caab58c-868a-4aff-b1da-36363ae721a4-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.763933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.763898 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5mkv\" (UniqueName: \"kubernetes.io/projected/4caab58c-868a-4aff-b1da-36363ae721a4-kube-api-access-f5mkv\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.980056 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:26.980025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:27.181762 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.181683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:27.471844 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.471813 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1074939-c496-4549-b730-b42efe4fb8a9" path="/var/lib/kubelet/pods/c1074939-c496-4549-b730-b42efe4fb8a9/volumes" Sep 16 14:00:27.472131 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.472115 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d84384-58ce-46ab-8b0a-ba00c7348bb5" path="/var/lib/kubelet/pods/f5d84384-58ce-46ab-8b0a-ba00c7348bb5/volumes" Sep 16 14:00:27.580931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.580901 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww" Sep 16 14:00:27.581342 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.580901 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8" Sep 16 14:00:27.609541 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.609478 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 14:00:27.610590 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.610561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:27.611570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.611551 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-733e5b-dc457b6f7-5nnww"] Sep 16 14:00:27.625149 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.625099 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 14:00:27.627671 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:27.627646 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-733e5b-6668b45cc8-85bp8"] Sep 16 14:00:28.097810 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:28.097774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:28.970613 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:28.970579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:29.473562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:29.473527 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caab58c-868a-4aff-b1da-36363ae721a4" path="/var/lib/kubelet/pods/4caab58c-868a-4aff-b1da-36363ae721a4/volumes" Sep 16 14:00:29.473947 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:29.473933 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" path="/var/lib/kubelet/pods/e2ca81e4-7908-49ab-a9e6-38da7f20bd9b/volumes" Sep 16 14:00:29.984358 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:29.984321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:30.468552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:30.468519 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:30.468703 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:30.468566 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:30.480987 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:30.480957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:30.608135 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:30.608108 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:00:30.943859 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:30.943826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:33.663510 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:33.663468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:33.788657 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:33.788616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:33.932595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:33.932509 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:34.104889 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:34.104846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:35.273526 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:35.273488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:35.374788 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:35.374750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:36.211762 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:36.211724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:36.318987 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:36.318949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:36.482399 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:36.482305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:37.361188 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:37.361143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:00:41.214484 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.214444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:41.290309 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.290276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:41.449384 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.449348 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:41.712113 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712074 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:00:41.712578 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712549 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caab58c-868a-4aff-b1da-36363ae721a4" containerName="payload-pre-processing" Sep 16 14:00:41.712578 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712574 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caab58c-868a-4aff-b1da-36363ae721a4" containerName="payload-pre-processing" Sep 16 14:00:41.712738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712614 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" containerName="payload-processing" Sep 16 14:00:41.712738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712624 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" containerName="payload-processing" Sep 16 14:00:41.712738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712722 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caab58c-868a-4aff-b1da-36363ae721a4" containerName="payload-pre-processing" Sep 16 14:00:41.712738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.712736 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ca81e4-7908-49ab-a9e6-38da7f20bd9b" containerName="payload-processing" Sep 16 14:00:41.716459 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.716437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.720724 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.720697 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 14:00:41.720852 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.720708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b5aadeb-openshift-default-dockercfg-nfn6b\"" Sep 16 14:00:41.729015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.728986 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:00:41.744189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.744138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:41.815271 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815271 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtmc\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815518 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815518 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815604 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815714 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815749 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.815795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.815742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.916803 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.916932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtmc\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917487 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917487 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.917896 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.917846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.919724 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.919700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.920086 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.920067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.927364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.927337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:41.927712 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:41.927683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtmc\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc\") pod \"e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:42.029713 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.029617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:42.311950 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.311919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:00:42.314160 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:42.314115 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7384b3_ee10_4829_9a63_257f15893321.slice/crio-61888a0a27163fcd119e5c0c7ca766e660a0485ebe44fc18e64b2a47a5e27660 WatchSource:0}: Error finding container 61888a0a27163fcd119e5c0c7ca766e660a0485ebe44fc18e64b2a47a5e27660: Status 404 returned error can't find the container with id 61888a0a27163fcd119e5c0c7ca766e660a0485ebe44fc18e64b2a47a5e27660 Sep 16 14:00:42.316846 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.316805 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:00:42.316951 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.316922 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:00:42.316997 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.316968 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:00:42.354891 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.354840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:42.679361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.679266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerStarted","Data":"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944"} Sep 16 14:00:42.679361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.679322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerStarted","Data":"61888a0a27163fcd119e5c0c7ca766e660a0485ebe44fc18e64b2a47a5e27660"} Sep 16 14:00:42.704349 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:42.704293 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podStartSLOduration=1.704272106 podStartE2EDuration="1.704272106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:42.702505218 +0000 UTC m=+1061.743099166" watchObservedRunningTime="2026-09-16 14:00:42.704272106 +0000 UTC m=+1061.744866061" Sep 16 14:00:43.029850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:43.029810 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:43.035769 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:43.035739 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:43.685274 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:43.685233 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:43.686337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:43.686316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:00:44.005261 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:44.005218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:44.726715 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:44.726673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:44.825839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:44.825802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:45.019467 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:45.019380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:45.988982 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:45.988937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:47.048878 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:47.048838 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:47.911539 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:47.911504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:48.831956 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:48.831921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:48.947976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:48.947944 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:49.386679 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:49.386647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:49.484262 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:49.484216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:49.629835 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:49.629797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:52.505482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:52.505448 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:53.030270 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.030233 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:00:53.034099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.034078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.036817 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.036796 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"e2e-conflict-b-b845ebcf-kserve-self-signed-certs\"" Sep 16 14:00:53.043670 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.043647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:00:53.134173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.134375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.134375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.134375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134261 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759gz\" (UniqueName: \"kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.134375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134346 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.134545 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.134383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235402 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759gz\" (UniqueName: \"kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235402 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235683 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235683 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235683 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235683 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235995 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.235995 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.235977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.236100 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.236004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.237799 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.237777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.237977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.237961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.244332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.244306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759gz\" (UniqueName: \"kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz\") pod \"e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.337068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.336973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:53.345943 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.345918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:00:53.619317 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.619286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:00:53.620853 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:00:53.620826 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9088cd2_e375_4bd5_b8b6_355e19c94458.slice/crio-cf00ad597530a2361e7f19cc80edb847eb56c7179f48db15027eec4a48180b11 WatchSource:0}: Error finding container cf00ad597530a2361e7f19cc80edb847eb56c7179f48db15027eec4a48180b11: Status 404 returned error can't find the container with id cf00ad597530a2361e7f19cc80edb847eb56c7179f48db15027eec4a48180b11 Sep 16 14:00:53.747508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.747464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" event={"ID":"b9088cd2-e375-4bd5-b8b6-355e19c94458","Type":"ContainerStarted","Data":"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f"} Sep 16 14:00:53.747668 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.747515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" event={"ID":"b9088cd2-e375-4bd5-b8b6-355e19c94458","Type":"ContainerStarted","Data":"cf00ad597530a2361e7f19cc80edb847eb56c7179f48db15027eec4a48180b11"} Sep 16 14:00:53.765791 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:00:53.765736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" podStartSLOduration=0.765720613 podStartE2EDuration="765.720613ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:53.76490894 +0000 UTC m=+1072.805502897" watchObservedRunningTime="2026-09-16 14:00:53.765720613 +0000 UTC m=+1072.806314569" Sep 16 14:01:03.346648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:03.346600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:01:03.346648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:03.346655 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:01:03.359968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:03.359934 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:01:03.822612 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:03.822572 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:01:04.818051 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.817943 2559 generic.go:356] "Generic (PLEG): container finished" podID="dd7384b3-ee10-4829-9a63-257f15893321" containerID="e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944" exitCode=0 Sep 16 14:01:04.818051 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.818024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerDied","Data":"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944"} Sep 16 14:01:04.818613 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.818521 2559 scope.go:117] "RemoveContainer" containerID="e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944" Sep 16 14:01:04.819633 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.819599 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:01:04.819728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.819671 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:01:04.819728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:04.819716 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:01:05.826746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:05.826708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerStarted","Data":"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853"} Sep 16 14:01:06.030490 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:06.030445 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:06.030753 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:06.030731 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 14:01:06.030801 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:06.030781 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 14:01:07.030386 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:07.030346 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 14:01:07.030775 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:07.030417 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 14:01:08.031016 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:08.030967 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 14:01:08.031415 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:08.031034 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 14:01:09.030493 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:09.030445 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 14:01:09.030677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:09.030509 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 14:01:09.453096 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:09.453050 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:01:10.030196 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:10.030153 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 16 14:01:10.030381 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:10.030212 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 16 14:01:11.043954 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:11.043917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:11.044363 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:11.044300 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:11.045146 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:11.045125 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:11.393100 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:11.393005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:12.314091 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:12.314057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:13.163783 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:13.163748 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:13.264851 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:13.264786 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:13.405280 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:13.405247 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:14.121468 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:14.121439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:14.682512 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:14.682475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:15.232113 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:15.232076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:15.659442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:15.659360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:15.756293 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:15.756260 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:15.856151 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:15.856119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:16.331507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:16.331474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:17.162210 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:17.162175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:19.052703 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:19.052657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:19.148906 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:19.148860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:22.098538 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.098499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:22.197989 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.197935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:22.480091 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.480053 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw"] Sep 16 14:01:22.484976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.484956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:22.491032 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.491003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:01:22.492361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.492159 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:01:22.500822 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.500795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw"] Sep 16 14:01:22.531150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.531113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw\" (UID: \"0d3d70f7-ae4f-4718-943b-c103e4163e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:22.632417 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.632377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw\" (UID: \"0d3d70f7-ae4f-4718-943b-c103e4163e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:22.633083 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.633060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw\" (UID: \"0d3d70f7-ae4f-4718-943b-c103e4163e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:22.796453 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:22.796361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:23.069707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.069678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw"] Sep 16 14:01:23.071509 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:01:23.071474 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3d70f7_ae4f_4718_943b_c103e4163e74.slice/crio-06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386 WatchSource:0}: Error finding container 06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386: Status 404 returned error can't find the container with id 06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386 Sep 16 14:01:23.826842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.826803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:23.930059 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.930023 2559 generic.go:356] "Generic (PLEG): container finished" podID="0d3d70f7-ae4f-4718-943b-c103e4163e74" containerID="188a47f9d017684991f2e151fe6239fbd48ce6563db6d176371ade219528fd4e" exitCode=0 Sep 16 14:01:23.930346 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.930062 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" event={"ID":"0d3d70f7-ae4f-4718-943b-c103e4163e74","Type":"ContainerDied","Data":"188a47f9d017684991f2e151fe6239fbd48ce6563db6d176371ade219528fd4e"} Sep 16 14:01:23.930490 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.930475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" event={"ID":"0d3d70f7-ae4f-4718-943b-c103e4163e74","Type":"ContainerStarted","Data":"06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386"} Sep 16 14:01:23.930632 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:23.930606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:24.710209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:24.710168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:25.072221 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.072193 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:25.161150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.161111 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca\") pod \"0d3d70f7-ae4f-4718-943b-c103e4163e74\" (UID: \"0d3d70f7-ae4f-4718-943b-c103e4163e74\") " Sep 16 14:01:25.161499 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.161474 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d3d70f7-ae4f-4718-943b-c103e4163e74" (UID: "0d3d70f7-ae4f-4718-943b-c103e4163e74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:01:25.262728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.262636 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d3d70f7-ae4f-4718-943b-c103e4163e74-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:25.636212 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.636129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:25.942928 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.942808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" event={"ID":"0d3d70f7-ae4f-4718-943b-c103e4163e74","Type":"ContainerDied","Data":"06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386"} Sep 16 14:01:25.942928 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.942827 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw" Sep 16 14:01:25.942928 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:25.942858 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b03725d67f216a405d7e74b0534032c4b563b65bf61642b90a6f7cc9b85386" Sep 16 14:01:26.568011 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:26.567976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:27.383658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:27.383620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:28.439750 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:28.439713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:30.469468 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:30.469433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:01:33.144533 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.144501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.233102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.233065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.320575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.320541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.337178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.337142 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:01:33.337609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.337594 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3d70f7-ae4f-4718-943b-c103e4163e74" containerName="revoke-keys" Sep 16 14:01:33.337661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.337613 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3d70f7-ae4f-4718-943b-c103e4163e74" containerName="revoke-keys" Sep 16 14:01:33.337709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.337699 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3d70f7-ae4f-4718-943b-c103e4163e74" containerName="revoke-keys" Sep 16 14:01:33.340948 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.340930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.343810 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.343790 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"distinct-w0-035c64-kserve-self-signed-certs\"" Sep 16 14:01:33.355338 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.355314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:01:33.401213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.401127 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw"] Sep 16 14:01:33.405178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.405144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.407648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.407619 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b5aadeb-f8jzw"] Sep 16 14:01:33.448090 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.448090 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kddh\" (UniqueName: \"kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.448363 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.448363 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.448453 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.448453 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.448416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.472091 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.472051 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3d70f7-ae4f-4718-943b-c103e4163e74" path="/var/lib/kubelet/pods/0d3d70f7-ae4f-4718-943b-c103e4163e74/volumes" Sep 16 14:01:33.507803 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.507751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.549877 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.549828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kddh\" (UniqueName: \"kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550059 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.549960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550059 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550059 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550239 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550239 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550357 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550534 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.550653 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.550505 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.552369 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.552345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.552537 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.552519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.560078 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.560047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kddh\" (UniqueName: \"kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh\") pod \"distinct-w0-035c64-kserve-59b6475ddd-9t659\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.650989 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.650942 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:33.768116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.768070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.868483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.868438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:33.930999 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.930975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:01:33.933229 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:01:33.933200 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35135d14_f3f3_4a3a_8562_bd3f04e38595.slice/crio-2576cb1fb2c24d18981d23ba9e9aac2e5e0ad1c30e3684e520fd0ead4a4a15b8 WatchSource:0}: Error finding container 2576cb1fb2c24d18981d23ba9e9aac2e5e0ad1c30e3684e520fd0ead4a4a15b8: Status 404 returned error can't find the container with id 2576cb1fb2c24d18981d23ba9e9aac2e5e0ad1c30e3684e520fd0ead4a4a15b8 Sep 16 14:01:33.991767 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:33.991734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" event={"ID":"35135d14-f3f3-4a3a-8562-bd3f04e38595","Type":"ContainerStarted","Data":"2576cb1fb2c24d18981d23ba9e9aac2e5e0ad1c30e3684e520fd0ead4a4a15b8"} Sep 16 14:01:34.166011 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:34.165973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:34.998792 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:34.998755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" event={"ID":"35135d14-f3f3-4a3a-8562-bd3f04e38595","Type":"ContainerStarted","Data":"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e"} Sep 16 14:01:35.031211 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:35.031160 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" podStartSLOduration=2.031143598 podStartE2EDuration="2.031143598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:35.029664109 +0000 UTC m=+1114.070258065" watchObservedRunningTime="2026-09-16 14:01:35.031143598 +0000 UTC m=+1114.071737545" Sep 16 14:01:36.213922 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:36.213864 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:36.311415 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:36.311378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:37.185613 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:37.185578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:37.865580 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:37.865547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:40.196524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:40.196488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:40.271174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:40.271134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:40.449696 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:40.449595 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:01:40.449896 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:40.449826 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" containerID="cri-o://30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853" gracePeriod=30 Sep 16 14:01:40.478818 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:40.478774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:41.047590 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:41.047552 2559 patch_prober.go:28] interesting pod/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 14:01:41.047763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:41.047624 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 14:01:41.568469 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:41.568432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:42.954125 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:42.954094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:43.651173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:43.651133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:43.651346 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:43.651191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:43.663889 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:43.663834 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:44.064628 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:44.064593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:01:45.711681 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.711613 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:45.873320 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873280 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873320 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873327 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873383 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873428 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873447 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873481 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxtmc\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873515 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873537 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873847 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873596 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token\") pod \"dd7384b3-ee10-4829-9a63-257f15893321\" (UID: \"dd7384b3-ee10-4829-9a63-257f15893321\") " Sep 16 14:01:45.873847 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:45.873847 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873759 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:45.873847 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873769 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:01:45.873847 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:45.874108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873947 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.874108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873962 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.874108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873977 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7384b3-ee10-4829-9a63-257f15893321-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.874108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873986 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.874108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.873992 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:45.875742 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.875716 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:01:45.875845 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.875827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:45.876308 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.876286 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:01:45.876440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.876322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc" (OuterVolumeSpecName: "kube-api-access-gxtmc") pod "dd7384b3-ee10-4829-9a63-257f15893321" (UID: "dd7384b3-ee10-4829-9a63-257f15893321"). InnerVolumeSpecName "kube-api-access-gxtmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:01:45.974654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.974563 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7384b3-ee10-4829-9a63-257f15893321-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.974654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.974598 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtmc\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-kube-api-access-gxtmc\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.974654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.974609 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.974654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.974619 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7384b3-ee10-4829-9a63-257f15893321-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:45.974654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:45.974626 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7384b3-ee10-4829-9a63-257f15893321-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:01:46.065597 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.065565 2559 generic.go:356] "Generic (PLEG): container finished" podID="dd7384b3-ee10-4829-9a63-257f15893321" containerID="30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853" exitCode=0 Sep 16 14:01:46.065795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.065615 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" Sep 16 14:01:46.065795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.065649 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerDied","Data":"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853"} Sep 16 14:01:46.065795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.065698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf" event={"ID":"dd7384b3-ee10-4829-9a63-257f15893321","Type":"ContainerDied","Data":"61888a0a27163fcd119e5c0c7ca766e660a0485ebe44fc18e64b2a47a5e27660"} Sep 16 14:01:46.065795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.065724 2559 scope.go:117] "RemoveContainer" containerID="30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853" Sep 16 14:01:46.075512 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.075492 2559 scope.go:117] "RemoveContainer" containerID="e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944" Sep 16 14:01:46.084313 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.084292 2559 scope.go:117] "RemoveContainer" containerID="30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853" Sep 16 14:01:46.084604 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:01:46.084584 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853\": container with ID starting with 30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853 not found: ID does not exist" containerID="30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853" Sep 16 14:01:46.084676 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.084617 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853"} err="failed to get container status \"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853\": rpc error: code = NotFound desc = could not find container \"30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853\": container with ID starting with 30c9807b4a9cc6ca5a114af8c85e589c7c9eab6ece349b8b1de5a80f71442853 not found: ID does not exist" Sep 16 14:01:46.084676 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.084641 2559 scope.go:117] "RemoveContainer" containerID="e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944" Sep 16 14:01:46.084926 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:01:46.084897 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944\": container with ID starting with e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944 not found: ID does not exist" containerID="e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944" Sep 16 14:01:46.084985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.084931 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944"} err="failed to get container status \"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944\": rpc error: code = NotFound desc = could not find container \"e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944\": container with ID starting with e31ce61cd60ef1cce110fe7606be2ba0706d67a156ccd283c61c8c95066c5944 not found: ID does not exist" Sep 16 14:01:46.089289 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.089263 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:01:46.094198 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:46.094172 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b5aadeb-openshift-default-cf4fbb6f6-hvfxf"] Sep 16 14:01:47.472375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:47.472344 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7384b3-ee10-4829-9a63-257f15893321" path="/var/lib/kubelet/pods/dd7384b3-ee10-4829-9a63-257f15893321/volumes" Sep 16 14:01:47.537711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:47.537677 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 14:01:47.566648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:47.566619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:47.945663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:47.945625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:48.045846 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:48.045812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:48.213154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:48.213069 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:48.340849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:48.340815 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:48.663114 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:48.663022 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:48.782314 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:48.782283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:49.025806 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:49.025768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:49.143861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:49.143822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:54.170535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:54.170501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:54.951702 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:54.951663 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:01:55.006962 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:55.006925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:55.094891 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:55.094834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:55.932821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:55.932789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:56.273242 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:56.273204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:56.601603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:56.601514 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:59.509718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:01:59.509683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:00.304639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:00.304597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:00.525520 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:00.525479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:01.713242 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:01.713205 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:02.556590 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.556549 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" podUID="25f0014b-37c2-4084-804f-33f62f74a7bc" containerName="main" containerID="cri-o://9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316" gracePeriod=45 Sep 16 14:02:02.799497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.799470 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 14:02:02.830907 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.830821 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8ptv\" (UniqueName: \"kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.830907 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.830886 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.831080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.830914 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.831080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.830978 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.831080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831015 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.831080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831038 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs\") pod \"25f0014b-37c2-4084-804f-33f62f74a7bc\" (UID: \"25f0014b-37c2-4084-804f-33f62f74a7bc\") " Sep 16 14:02:02.831281 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:02.831281 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831257 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home" (OuterVolumeSpecName: "home") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:02.831398 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831278 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:02.831455 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831400 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:02.831455 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831415 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:02.831455 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.831423 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:02.833138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.833103 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm" (OuterVolumeSpecName: "dshm") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:02.833255 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.833179 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:02.833255 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.833186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv" (OuterVolumeSpecName: "kube-api-access-f8ptv") pod "25f0014b-37c2-4084-804f-33f62f74a7bc" (UID: "25f0014b-37c2-4084-804f-33f62f74a7bc"). InnerVolumeSpecName "kube-api-access-f8ptv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:02.932850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.932802 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ptv\" (UniqueName: \"kubernetes.io/projected/25f0014b-37c2-4084-804f-33f62f74a7bc-kube-api-access-f8ptv\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:02.932850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.932842 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25f0014b-37c2-4084-804f-33f62f74a7bc-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:02.932850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:02.932854 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f0014b-37c2-4084-804f-33f62f74a7bc-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:03.158659 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.158574 2559 generic.go:356] "Generic (PLEG): container finished" podID="25f0014b-37c2-4084-804f-33f62f74a7bc" containerID="9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316" exitCode=0 Sep 16 14:02:03.158659 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.158616 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" Sep 16 14:02:03.158659 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.158626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" event={"ID":"25f0014b-37c2-4084-804f-33f62f74a7bc","Type":"ContainerDied","Data":"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316"} Sep 16 14:02:03.158659 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.158658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8" event={"ID":"25f0014b-37c2-4084-804f-33f62f74a7bc","Type":"ContainerDied","Data":"e0f33904a83c8ef140f71ca4d841395f2639c13ed7b484074614ccbaa5376bcc"} Sep 16 14:02:03.158948 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.158676 2559 scope.go:117] "RemoveContainer" containerID="9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316" Sep 16 14:02:03.168033 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.168013 2559 scope.go:117] "RemoveContainer" containerID="9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316" Sep 16 14:02:03.168279 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:03.168261 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316\": container with ID starting with 9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316 not found: ID does not exist" containerID="9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316" Sep 16 14:02:03.168326 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.168288 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316"} err="failed to get container status \"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316\": rpc error: code = NotFound desc = could not find container \"9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316\": container with ID starting with 9954727065a0832add3718207f03f426b15131e97944e63db8d2c5772fce1316 not found: ID does not exist" Sep 16 14:02:03.185454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.185423 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 14:02:03.187322 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.187302 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b52a89/test-model-b52a89-kserve-85dfb984bc-xpdr8"] Sep 16 14:02:03.472015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.471981 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25f0014b-37c2-4084-804f-33f62f74a7bc" path="/var/lib/kubelet/pods/25f0014b-37c2-4084-804f-33f62f74a7bc/volumes" Sep 16 14:02:03.472304 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472289 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c"] Sep 16 14:02:03.472673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472661 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f0014b-37c2-4084-804f-33f62f74a7bc" containerName="main" Sep 16 14:02:03.472709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472676 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f0014b-37c2-4084-804f-33f62f74a7bc" containerName="main" Sep 16 14:02:03.472709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472691 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:03.472709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472697 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:03.472804 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472776 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:03.472804 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472784 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:03.472804 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.472792 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f0014b-37c2-4084-804f-33f62f74a7bc" containerName="main" Sep 16 14:02:03.476972 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.476956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:03.479834 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.479820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:02:03.479899 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.479857 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:02:03.485822 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.485795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c"] Sep 16 14:02:03.537682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.537645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c\" (UID: \"3b547847-4ddd-4a5a-b999-cedff4f89dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:03.639055 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.639017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c\" (UID: \"3b547847-4ddd-4a5a-b999-cedff4f89dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:03.639641 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.639621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c\" (UID: \"3b547847-4ddd-4a5a-b999-cedff4f89dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:03.788573 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:03.788486 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:04.067008 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.066934 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c"] Sep 16 14:02:04.072432 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:04.072397 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b547847_4ddd_4a5a_b999_cedff4f89dfc.slice/crio-1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172 WatchSource:0}: Error finding container 1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172: Status 404 returned error can't find the container with id 1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172 Sep 16 14:02:04.144080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.144046 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:02:04.168204 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.168176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" event={"ID":"3b547847-4ddd-4a5a-b999-cedff4f89dfc","Type":"ContainerStarted","Data":"1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172"} Sep 16 14:02:04.256509 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.256474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:04.355681 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.355596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:04.565300 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.565235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:04.681454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.681371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:04.784434 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.784399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:04.903961 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.903922 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:02:04.904357 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.904345 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:04.904444 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.904359 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7384b3-ee10-4829-9a63-257f15893321" containerName="istio-proxy" Sep 16 14:02:04.907402 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.907387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:04.909658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.909632 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-740a25-dockercfg-wkpz8\"" Sep 16 14:02:04.910205 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.910186 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-740a25\"" Sep 16 14:02:04.918542 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.918519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:02:04.952248 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.952221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:04.952399 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:04.952372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvb4l\" (UniqueName: \"kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.053158 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.053113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvb4l\" (UniqueName: \"kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.053333 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.053170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.053803 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.053782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.063052 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.063021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvb4l\" (UniqueName: \"kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l\") pod \"payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.137340 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.137295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:05.175717 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.175676 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b547847-4ddd-4a5a-b999-cedff4f89dfc" containerID="c923ef3232456b33a3b7873b331498bb40b127f03e0a5a6cfbce3530f50c8dec" exitCode=0 Sep 16 14:02:05.176115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.175716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" event={"ID":"3b547847-4ddd-4a5a-b999-cedff4f89dfc","Type":"ContainerDied","Data":"c923ef3232456b33a3b7873b331498bb40b127f03e0a5a6cfbce3530f50c8dec"} Sep 16 14:02:05.233724 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.233637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:05.507325 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:05.507248 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:02:05.510619 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:05.510585 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9bc507_b2e6_4ce0_92bd_0613672e8b88.slice/crio-e346ffa71cd86c32f228f981b5ad6050c01a05cb0ec0f043a68e7f0e55fbddd2 WatchSource:0}: Error finding container e346ffa71cd86c32f228f981b5ad6050c01a05cb0ec0f043a68e7f0e55fbddd2: Status 404 returned error can't find the container with id e346ffa71cd86c32f228f981b5ad6050c01a05cb0ec0f043a68e7f0e55fbddd2 Sep 16 14:02:06.079508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.079474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:06.183502 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.183463 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" event={"ID":"cc9bc507-b2e6-4ce0-92bd-0613672e8b88","Type":"ContainerStarted","Data":"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0"} Sep 16 14:02:06.183502 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.183521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" event={"ID":"cc9bc507-b2e6-4ce0-92bd-0613672e8b88","Type":"ContainerStarted","Data":"e346ffa71cd86c32f228f981b5ad6050c01a05cb0ec0f043a68e7f0e55fbddd2"} Sep 16 14:02:06.184039 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.183545 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:06.205370 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.205305 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" podStartSLOduration=2.205282864 podStartE2EDuration="2.205282864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:06.202795584 +0000 UTC m=+1145.243389538" watchObservedRunningTime="2026-09-16 14:02:06.205282864 +0000 UTC m=+1145.245876819" Sep 16 14:02:06.318739 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.318708 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:06.365450 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.365355 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca\") pod \"3b547847-4ddd-4a5a-b999-cedff4f89dfc\" (UID: \"3b547847-4ddd-4a5a-b999-cedff4f89dfc\") " Sep 16 14:02:06.365769 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.365741 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b547847-4ddd-4a5a-b999-cedff4f89dfc" (UID: "3b547847-4ddd-4a5a-b999-cedff4f89dfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:06.466221 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.466189 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b547847-4ddd-4a5a-b999-cedff4f89dfc-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:06.960949 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:06.960880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:07.191095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:07.191062 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" Sep 16 14:02:07.191462 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:07.191097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c" event={"ID":"3b547847-4ddd-4a5a-b999-cedff4f89dfc","Type":"ContainerDied","Data":"1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172"} Sep 16 14:02:07.191462 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:07.191142 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e21b50f4c26ba4107b1e6529429430f014db6e2e1a2eaadf864b5830cdbf172" Sep 16 14:02:07.935906 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:07.935835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:08.994476 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:08.994432 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:02:10.237371 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:10.237332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:10.656163 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:10.656076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:13.626052 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:13.626013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:14.609998 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.609964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:14.662766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.662723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 14:02:14.663201 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.662972 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" podUID="12b11736-4bf1-4536-9914-1434c18c7533" containerName="payload-pre-processing" containerID="cri-o://a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638" gracePeriod=30 Sep 16 14:02:14.866222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.866138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:14.876166 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.876139 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c"] Sep 16 14:02:14.881040 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.881017 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b52a89-qb55c"] Sep 16 14:02:14.916635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.916612 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 14:02:14.966800 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:14.966765 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:15.044764 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.044734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume\") pod \"12b11736-4bf1-4536-9914-1434c18c7533\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " Sep 16 14:02:15.044962 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.044839 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdmmc\" (UniqueName: \"kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc\") pod \"12b11736-4bf1-4536-9914-1434c18c7533\" (UID: \"12b11736-4bf1-4536-9914-1434c18c7533\") " Sep 16 14:02:15.045148 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.045126 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12b11736-4bf1-4536-9914-1434c18c7533" (UID: "12b11736-4bf1-4536-9914-1434c18c7533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:15.046910 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.046863 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc" (OuterVolumeSpecName: "kube-api-access-fdmmc") pod "12b11736-4bf1-4536-9914-1434c18c7533" (UID: "12b11736-4bf1-4536-9914-1434c18c7533"). InnerVolumeSpecName "kube-api-access-fdmmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:15.146401 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.146326 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmmc\" (UniqueName: \"kubernetes.io/projected/12b11736-4bf1-4536-9914-1434c18c7533-kube-api-access-fdmmc\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:15.146401 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.146355 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b11736-4bf1-4536-9914-1434c18c7533-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:15.236071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.236040 2559 generic.go:356] "Generic (PLEG): container finished" podID="12b11736-4bf1-4536-9914-1434c18c7533" containerID="a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638" exitCode=0 Sep 16 14:02:15.236247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.236083 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" Sep 16 14:02:15.236247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.236127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" event={"ID":"12b11736-4bf1-4536-9914-1434c18c7533","Type":"ContainerDied","Data":"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638"} Sep 16 14:02:15.236247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.236175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh" event={"ID":"12b11736-4bf1-4536-9914-1434c18c7533","Type":"ContainerDied","Data":"b744e3cd4b6b94179cfb12b4d7de705c389a06ae4a83f1ddd3dc0c61ebb8bba8"} Sep 16 14:02:15.236247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.236196 2559 scope.go:117] "RemoveContainer" containerID="a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638" Sep 16 14:02:15.245976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.245927 2559 scope.go:117] "RemoveContainer" containerID="a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638" Sep 16 14:02:15.246219 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:15.246201 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638\": container with ID starting with a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638 not found: ID does not exist" containerID="a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638" Sep 16 14:02:15.246263 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.246227 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638"} err="failed to get container status \"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638\": rpc error: code = NotFound desc = could not find container \"a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638\": container with ID starting with a694eccc42d19524219c15787e3dab67565ac78ecc82667571a76cae4b0a4638 not found: ID does not exist" Sep 16 14:02:15.260976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.260957 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 14:02:15.270586 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.270559 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b52a89-675568f578-hm7rh"] Sep 16 14:02:15.277082 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.277060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:15.373243 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.373208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:15.472180 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.472146 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b11736-4bf1-4536-9914-1434c18c7533" path="/var/lib/kubelet/pods/12b11736-4bf1-4536-9914-1434c18c7533/volumes" Sep 16 14:02:15.472575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:15.472556 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b547847-4ddd-4a5a-b999-cedff4f89dfc" path="/var/lib/kubelet/pods/3b547847-4ddd-4a5a-b999-cedff4f89dfc/volumes" Sep 16 14:02:16.386524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:16.386481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:17.193682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:17.193652 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:02:17.334535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:17.334501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:18.299033 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:18.299001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:19.164537 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.164496 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" podUID="b9088cd2-e375-4bd5-b8b6-355e19c94458" containerName="main" containerID="cri-o://7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f" gracePeriod=45 Sep 16 14:02:19.399098 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.399065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:19.416635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.416578 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:02:19.490692 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490658 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.490904 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490717 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.490904 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490757 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.490904 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490899 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759gz\" (UniqueName: \"kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.491080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.491080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490948 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home" (OuterVolumeSpecName: "home") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:19.491080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490961 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:19.491080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.490979 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache\") pod \"b9088cd2-e375-4bd5-b8b6-355e19c94458\" (UID: \"b9088cd2-e375-4bd5-b8b6-355e19c94458\") " Sep 16 14:02:19.491260 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.491184 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:19.491330 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.491299 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:19.491367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.491336 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:19.491367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.491348 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:19.493034 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.493002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz" (OuterVolumeSpecName: "kube-api-access-759gz") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "kube-api-access-759gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:19.493034 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.493002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:19.493156 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.493044 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm" (OuterVolumeSpecName: "dshm") pod "b9088cd2-e375-4bd5-b8b6-355e19c94458" (UID: "b9088cd2-e375-4bd5-b8b6-355e19c94458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:19.592831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.592791 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9088cd2-e375-4bd5-b8b6-355e19c94458-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:19.592831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.592825 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759gz\" (UniqueName: \"kubernetes.io/projected/b9088cd2-e375-4bd5-b8b6-355e19c94458-kube-api-access-759gz\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:19.593056 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:19.592840 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9088cd2-e375-4bd5-b8b6-355e19c94458-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:20.266291 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.266254 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9088cd2-e375-4bd5-b8b6-355e19c94458" containerID="7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f" exitCode=0 Sep 16 14:02:20.266470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.266300 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" Sep 16 14:02:20.266470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.266336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" event={"ID":"b9088cd2-e375-4bd5-b8b6-355e19c94458","Type":"ContainerDied","Data":"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f"} Sep 16 14:02:20.266470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.266379 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d" event={"ID":"b9088cd2-e375-4bd5-b8b6-355e19c94458","Type":"ContainerDied","Data":"cf00ad597530a2361e7f19cc80edb847eb56c7179f48db15027eec4a48180b11"} Sep 16 14:02:20.266470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.266396 2559 scope.go:117] "RemoveContainer" containerID="7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f" Sep 16 14:02:20.276107 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.276088 2559 scope.go:117] "RemoveContainer" containerID="7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f" Sep 16 14:02:20.276361 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:20.276344 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f\": container with ID starting with 7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f not found: ID does not exist" containerID="7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f" Sep 16 14:02:20.276434 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.276369 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f"} err="failed to get container status \"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f\": rpc error: code = NotFound desc = could not find container \"7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f\": container with ID starting with 7fbb55910f68cf257ce995dcbadafa573b0bc2f629407a1f685dfecb36f99a4f not found: ID does not exist" Sep 16 14:02:20.304737 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.304702 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:02:20.311150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:20.311124 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-b-b845ebcf-kserve-6b4cf89c94-cql7d"] Sep 16 14:02:21.212807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:21.212775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:21.474412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:21.474332 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9088cd2-e375-4bd5-b8b6-355e19c94458" path="/var/lib/kubelet/pods/b9088cd2-e375-4bd5-b8b6-355e19c94458/volumes" Sep 16 14:02:22.113247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:22.113208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:22.218707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:22.218660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:22.350909 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:22.350851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:22.586932 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:22.586899 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:24.000373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.000279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:24.012999 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.012968 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" podUID="8101a30e-0041-48f5-816c-709de955412a" containerName="main" containerID="cri-o://cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c" gracePeriod=45 Sep 16 14:02:24.118121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.118092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:24.264839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.264778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:02:24.296651 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.296618 2559 generic.go:356] "Generic (PLEG): container finished" podID="8101a30e-0041-48f5-816c-709de955412a" containerID="cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c" exitCode=0 Sep 16 14:02:24.296842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.296660 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" Sep 16 14:02:24.296842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.296708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" event={"ID":"8101a30e-0041-48f5-816c-709de955412a","Type":"ContainerDied","Data":"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c"} Sep 16 14:02:24.296842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.296751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl" event={"ID":"8101a30e-0041-48f5-816c-709de955412a","Type":"ContainerDied","Data":"c2306a3a03f0801b4dcac2cd8b1eb03d4b0fb4dc132cabf9bfb59d9b9c85c4cc"} Sep 16 14:02:24.296842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.296769 2559 scope.go:117] "RemoveContainer" containerID="cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c" Sep 16 14:02:24.307573 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.307553 2559 scope.go:117] "RemoveContainer" containerID="cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c" Sep 16 14:02:24.307949 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:24.307924 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c\": container with ID starting with cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c not found: ID does not exist" containerID="cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c" Sep 16 14:02:24.308024 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.307961 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c"} err="failed to get container status \"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c\": rpc error: code = NotFound desc = could not find container \"cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c\": container with ID starting with cd68e61f6d75d3262a853f0bb3caad105b007c9c77cd520d6be017ed8fc2ee7c not found: ID does not exist" Sep 16 14:02:24.339466 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339437 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339507 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339540 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb6zm\" (UniqueName: \"kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339565 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339665 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir\") pod \"8101a30e-0041-48f5-816c-709de955412a\" (UID: \"8101a30e-0041-48f5-816c-709de955412a\") " Sep 16 14:02:24.339849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home" (OuterVolumeSpecName: "home") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:24.339988 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339933 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:24.339988 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.339960 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:24.340099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.340079 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.340154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.340103 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.340154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.340116 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.341718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.341679 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:24.341809 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.341748 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm" (OuterVolumeSpecName: "dshm") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:02:24.341809 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.341762 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm" (OuterVolumeSpecName: "kube-api-access-qb6zm") pod "8101a30e-0041-48f5-816c-709de955412a" (UID: "8101a30e-0041-48f5-816c-709de955412a"). InnerVolumeSpecName "kube-api-access-qb6zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:24.359322 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.359283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:24.441677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.441635 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6zm\" (UniqueName: \"kubernetes.io/projected/8101a30e-0041-48f5-816c-709de955412a-kube-api-access-qb6zm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.441677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.441670 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101a30e-0041-48f5-816c-709de955412a-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.441677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.441680 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8101a30e-0041-48f5-816c-709de955412a-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:24.457755 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.457723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:24.625364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.625330 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:02:24.628466 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.628437 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/e2e-conflict-a-b845ebcf-kserve-76d79488-msjkl"] Sep 16 14:02:24.690505 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:24.690472 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:25.053435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:25.053398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:25.473726 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:25.473681 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8101a30e-0041-48f5-816c-709de955412a" path="/var/lib/kubelet/pods/8101a30e-0041-48f5-816c-709de955412a/volumes" Sep 16 14:02:25.512240 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:25.512201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:25.817465 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:25.817373 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:26.435301 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:26.435261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:27.843268 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:27.843229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:28.091933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:28.091900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:28.859638 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:28.859598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:28.977073 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:28.977038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:30.589133 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:30.589102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:30.707617 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:30.707587 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:31.610764 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:31.610724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:32.508424 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:32.508382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:32.556917 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:32.556880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:02:33.520489 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.520436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:33.545379 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.545347 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 14:02:33.545580 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.545554 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" podUID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" containerName="payload-processing" containerID="cri-o://afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4" gracePeriod=30 Sep 16 14:02:33.556083 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.556055 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 14:02:33.556463 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.556418 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" podUID="dd0d306e-7153-4867-bf21-44c0533553d5" containerName="payload-pre-processing" containerID="cri-o://75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769" gracePeriod=30 Sep 16 14:02:33.630696 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.630662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:33.743070 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.743028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:33.834464 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.834438 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 14:02:33.837280 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.837261 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 14:02:33.870964 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.870934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:33.934941 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.934904 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn6tn\" (UniqueName: \"kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn\") pod \"dd0d306e-7153-4867-bf21-44c0533553d5\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " Sep 16 14:02:33.935131 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.935079 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq2f6\" (UniqueName: \"kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6\") pod \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " Sep 16 14:02:33.935131 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.935107 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume\") pod \"dd0d306e-7153-4867-bf21-44c0533553d5\" (UID: \"dd0d306e-7153-4867-bf21-44c0533553d5\") " Sep 16 14:02:33.935259 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.935133 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume\") pod \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\" (UID: \"925f7d04-83f3-4b19-a3f2-ef60c3af5de1\") " Sep 16 14:02:33.935501 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.935474 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "925f7d04-83f3-4b19-a3f2-ef60c3af5de1" (UID: "925f7d04-83f3-4b19-a3f2-ef60c3af5de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:33.935501 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.935488 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd0d306e-7153-4867-bf21-44c0533553d5" (UID: "dd0d306e-7153-4867-bf21-44c0533553d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:33.937034 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.937006 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn" (OuterVolumeSpecName: "kube-api-access-wn6tn") pod "dd0d306e-7153-4867-bf21-44c0533553d5" (UID: "dd0d306e-7153-4867-bf21-44c0533553d5"). InnerVolumeSpecName "kube-api-access-wn6tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:33.937224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.937207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6" (OuterVolumeSpecName: "kube-api-access-nq2f6") pod "925f7d04-83f3-4b19-a3f2-ef60c3af5de1" (UID: "925f7d04-83f3-4b19-a3f2-ef60c3af5de1"). InnerVolumeSpecName "kube-api-access-nq2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:33.976152 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:33.976121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:34.036351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.036309 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2f6\" (UniqueName: \"kubernetes.io/projected/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-kube-api-access-nq2f6\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:34.036351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.036344 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0d306e-7153-4867-bf21-44c0533553d5-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:34.036351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.036357 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7d04-83f3-4b19-a3f2-ef60c3af5de1-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:34.036577 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.036370 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn6tn\" (UniqueName: \"kubernetes.io/projected/dd0d306e-7153-4867-bf21-44c0533553d5-kube-api-access-wn6tn\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:02:34.354462 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.354426 2559 generic.go:356] "Generic (PLEG): container finished" podID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" containerID="afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4" exitCode=0 Sep 16 14:02:34.354758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.354467 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" Sep 16 14:02:34.354758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.354475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" event={"ID":"925f7d04-83f3-4b19-a3f2-ef60c3af5de1","Type":"ContainerDied","Data":"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4"} Sep 16 14:02:34.354758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.354533 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl" event={"ID":"925f7d04-83f3-4b19-a3f2-ef60c3af5de1","Type":"ContainerDied","Data":"7bac3221df427388adfc9829a43d49141be4f8633be8dc363475c21445090e70"} Sep 16 14:02:34.354758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.354554 2559 scope.go:117] "RemoveContainer" containerID="afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4" Sep 16 14:02:34.356029 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.355993 2559 generic.go:356] "Generic (PLEG): container finished" podID="dd0d306e-7153-4867-bf21-44c0533553d5" containerID="75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769" exitCode=0 Sep 16 14:02:34.356142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.356028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" event={"ID":"dd0d306e-7153-4867-bf21-44c0533553d5","Type":"ContainerDied","Data":"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769"} Sep 16 14:02:34.356142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.356039 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" Sep 16 14:02:34.356142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.356072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4" event={"ID":"dd0d306e-7153-4867-bf21-44c0533553d5","Type":"ContainerDied","Data":"43d30c342cdcd7ef4665a60f098c8a6bfbcc86f9de4c65f99f86c1c342f9cc80"} Sep 16 14:02:34.365732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.365697 2559 scope.go:117] "RemoveContainer" containerID="afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4" Sep 16 14:02:34.367926 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:34.366184 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4\": container with ID starting with afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4 not found: ID does not exist" containerID="afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4" Sep 16 14:02:34.367926 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.366212 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4"} err="failed to get container status \"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4\": rpc error: code = NotFound desc = could not find container \"afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4\": container with ID starting with afb5371d220bd1ad29cd02d142e8a3ebe69351b2c6c8c1903d0db76652b068c4 not found: ID does not exist" Sep 16 14:02:34.367926 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.366231 2559 scope.go:117] "RemoveContainer" containerID="75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769" Sep 16 14:02:34.374853 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.374836 2559 scope.go:117] "RemoveContainer" containerID="75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769" Sep 16 14:02:34.375131 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:34.375114 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769\": container with ID starting with 75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769 not found: ID does not exist" containerID="75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769" Sep 16 14:02:34.375176 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.375138 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769"} err="failed to get container status \"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769\": rpc error: code = NotFound desc = could not find container \"75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769\": container with ID starting with 75213530473d1fb596eb60dd412a5a4cc671b9a78dbfc618e677c5996603c769 not found: ID does not exist" Sep 16 14:02:34.384912 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.384886 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 14:02:34.390164 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.390143 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-95743e-6c8884f49-mg6xl"] Sep 16 14:02:34.405194 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.405174 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 14:02:34.407916 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.407894 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-95743e-fd558d6c8-4xrg4"] Sep 16 14:02:34.844882 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.844826 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:02:34.845377 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845354 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8101a30e-0041-48f5-816c-709de955412a" containerName="main" Sep 16 14:02:34.845447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845381 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8101a30e-0041-48f5-816c-709de955412a" containerName="main" Sep 16 14:02:34.845447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845404 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b11736-4bf1-4536-9914-1434c18c7533" containerName="payload-pre-processing" Sep 16 14:02:34.845447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845413 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b11736-4bf1-4536-9914-1434c18c7533" containerName="payload-pre-processing" Sep 16 14:02:34.845447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845432 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b547847-4ddd-4a5a-b999-cedff4f89dfc" containerName="revoke-keys" Sep 16 14:02:34.845447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845441 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b547847-4ddd-4a5a-b999-cedff4f89dfc" containerName="revoke-keys" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845465 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9088cd2-e375-4bd5-b8b6-355e19c94458" containerName="main" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845473 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9088cd2-e375-4bd5-b8b6-355e19c94458" containerName="main" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845484 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0d306e-7153-4867-bf21-44c0533553d5" containerName="payload-pre-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845493 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0d306e-7153-4867-bf21-44c0533553d5" containerName="payload-pre-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845503 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" containerName="payload-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845512 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" containerName="payload-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845612 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" containerName="payload-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845623 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b547847-4ddd-4a5a-b999-cedff4f89dfc" containerName="revoke-keys" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845636 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b11736-4bf1-4536-9914-1434c18c7533" containerName="payload-pre-processing" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845646 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8101a30e-0041-48f5-816c-709de955412a" containerName="main" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845655 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9088cd2-e375-4bd5-b8b6-355e19c94458" containerName="main" Sep 16 14:02:34.845675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.845663 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0d306e-7153-4867-bf21-44c0533553d5" containerName="payload-pre-processing" Sep 16 14:02:34.850224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.850203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:34.855156 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.855136 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9187f5\"" Sep 16 14:02:34.855247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.855146 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9187f5-dockercfg-pv4bb\"" Sep 16 14:02:34.869369 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.869328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:02:34.913567 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.913528 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:02:34.916828 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.916811 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:34.940471 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.940442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:02:34.943195 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.943169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:34.943297 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.943219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:34.943364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.943294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pmq\" (UniqueName: \"kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:34.943429 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:34.943412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ch8\" (UniqueName: \"kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.043900 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.043838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pmq\" (UniqueName: \"kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:35.044079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.043953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ch8\" (UniqueName: \"kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.044079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.043989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.044079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.044015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:35.044720 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.044698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.044767 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.044709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:35.068040 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.068011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ch8\" (UniqueName: \"kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8\") pod \"payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.073655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.073631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pmq\" (UniqueName: \"kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq\") pod \"payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:35.160757 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.160670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:35.190085 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.190058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:35.226280 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.226195 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:35.306690 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.306654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:35.447526 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.447498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:02:35.449622 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:35.449595 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176046fb_eff3_486c_8ed0_7bc51e3b2184.slice/crio-d7ed6dc614694e43aafbd6dfec92a0aa12523cc4496dec7adc804953c456ca31 WatchSource:0}: Error finding container d7ed6dc614694e43aafbd6dfec92a0aa12523cc4496dec7adc804953c456ca31: Status 404 returned error can't find the container with id d7ed6dc614694e43aafbd6dfec92a0aa12523cc4496dec7adc804953c456ca31 Sep 16 14:02:35.474118 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.474085 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925f7d04-83f3-4b19-a3f2-ef60c3af5de1" path="/var/lib/kubelet/pods/925f7d04-83f3-4b19-a3f2-ef60c3af5de1/volumes" Sep 16 14:02:35.474578 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.474563 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd0d306e-7153-4867-bf21-44c0533553d5" path="/var/lib/kubelet/pods/dd0d306e-7153-4867-bf21-44c0533553d5/volumes" Sep 16 14:02:35.516791 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.516762 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:02:35.519059 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:35.519032 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6010bdfc_489d_4379_816a_de3b6a2a441c.slice/crio-347946a1e093ee6b795dd2cc59eda46c040ed12bd1a0588d3327201373ca8724 WatchSource:0}: Error finding container 347946a1e093ee6b795dd2cc59eda46c040ed12bd1a0588d3327201373ca8724: Status 404 returned error can't find the container with id 347946a1e093ee6b795dd2cc59eda46c040ed12bd1a0588d3327201373ca8724 Sep 16 14:02:35.549696 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:35.549672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:36.372416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.372363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" event={"ID":"6010bdfc-489d-4379-816a-de3b6a2a441c","Type":"ContainerStarted","Data":"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c"} Sep 16 14:02:36.372930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.372433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" event={"ID":"6010bdfc-489d-4379-816a-de3b6a2a441c","Type":"ContainerStarted","Data":"347946a1e093ee6b795dd2cc59eda46c040ed12bd1a0588d3327201373ca8724"} Sep 16 14:02:36.372930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.372465 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:36.374175 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.374147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" event={"ID":"176046fb-eff3-486c-8ed0-7bc51e3b2184","Type":"ContainerStarted","Data":"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3"} Sep 16 14:02:36.374275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.374194 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:36.374275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.374217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" event={"ID":"176046fb-eff3-486c-8ed0-7bc51e3b2184","Type":"ContainerStarted","Data":"d7ed6dc614694e43aafbd6dfec92a0aa12523cc4496dec7adc804953c456ca31"} Sep 16 14:02:36.396219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.396165 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" podStartSLOduration=2.396149959 podStartE2EDuration="2.396149959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:36.394308676 +0000 UTC m=+1175.434902631" watchObservedRunningTime="2026-09-16 14:02:36.396149959 +0000 UTC m=+1175.436743912" Sep 16 14:02:36.420688 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.420619 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" podStartSLOduration=2.42059765 podStartE2EDuration="2.42059765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:36.420555154 +0000 UTC m=+1175.461149107" watchObservedRunningTime="2026-09-16 14:02:36.42059765 +0000 UTC m=+1175.461191604" Sep 16 14:02:36.494222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:36.494187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:37.494640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:37.494609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:38.404977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:38.404943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:39.472850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:39.472812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:41.937361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:41.937329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:42.021288 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:42.021254 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:42.234971 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:42.234936 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:42.826323 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:42.826289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:43.159464 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:43.159370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:43.544953 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:43.544919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:46.093485 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:46.093450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:46.754365 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:46.754325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:46.915141 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:46.915107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:47.144498 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.144412 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:02:47.148118 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.148100 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.154736 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.154711 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"distinct-w4-19203d-kserve-self-signed-certs\"" Sep 16 14:02:47.183269 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.183239 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:02:47.257263 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.257449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257274 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wts7b\" (UniqueName: \"kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.257449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257349 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.257449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.257449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.257449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.257446 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.358992 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.358956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wts7b\" (UniqueName: \"kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359514 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359514 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.359591 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.359523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.361215 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.361193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.361589 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.361569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.370951 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.370929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wts7b\" (UniqueName: \"kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b\") pod \"distinct-w4-19203d-kserve-97d9d56d-78hqp\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.381513 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.381489 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:02:47.381616 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.381570 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:02:47.457135 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.457099 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:47.472286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.472207 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:47.608859 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.608827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:47.733967 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:47.733940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:02:47.735669 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:47.735640 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09aae84e_476b_410d_98f9_86f02fc5823b.slice/crio-a9a9da488659250afd7462a5643f50552b6f040c9076bfdd303a98f9ade535d5 WatchSource:0}: Error finding container a9a9da488659250afd7462a5643f50552b6f040c9076bfdd303a98f9ade535d5: Status 404 returned error can't find the container with id a9a9da488659250afd7462a5643f50552b6f040c9076bfdd303a98f9ade535d5 Sep 16 14:02:48.444219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:48.444178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" event={"ID":"09aae84e-476b-410d-98f9-86f02fc5823b","Type":"ContainerStarted","Data":"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400"} Sep 16 14:02:48.444219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:48.444230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" event={"ID":"09aae84e-476b-410d-98f9-86f02fc5823b","Type":"ContainerStarted","Data":"a9a9da488659250afd7462a5643f50552b6f040c9076bfdd303a98f9ade535d5"} Sep 16 14:02:48.466226 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:48.466134 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" podStartSLOduration=1.466113936 podStartE2EDuration="1.466113936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:48.465043213 +0000 UTC m=+1187.505637168" watchObservedRunningTime="2026-09-16 14:02:48.466113936 +0000 UTC m=+1187.506707891" Sep 16 14:02:48.859112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:48.859027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:49.788360 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:49.788321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:50.726961 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:50.726921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:51.627609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:51.627578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:51.729161 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:51.729112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:52.170192 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:52.170145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:52.269655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:52.269625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:52.441247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:52.441158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:52.733116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:52.733083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:53.621481 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:53.621445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:53.822375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:53.822338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:54.381515 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.381482 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-de0b3e"] Sep 16 14:02:54.385096 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.385080 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:54.388005 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.387978 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:02:54.390918 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.390894 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-de0b3e"] Sep 16 14:02:54.532023 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.531975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj99\" (UniqueName: \"kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99\") pod \"test-curl-7896-de0b3e\" (UID: \"8bb8a784-0ad6-4da2-b204-ae5b50c07676\") " pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:54.632937 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.632815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj99\" (UniqueName: \"kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99\") pod \"test-curl-7896-de0b3e\" (UID: \"8bb8a784-0ad6-4da2-b204-ae5b50c07676\") " pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:54.642760 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.642723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj99\" (UniqueName: \"kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99\") pod \"test-curl-7896-de0b3e\" (UID: \"8bb8a784-0ad6-4da2-b204-ae5b50c07676\") " pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:54.695808 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.695768 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:54.732440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.732399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:54.965509 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:54.965429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-de0b3e"] Sep 16 14:02:54.967951 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:02:54.967928 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb8a784_0ad6_4da2_b204_ae5b50c07676.slice/crio-aeb362ec1e41ee66286583ff027a94c24685c0644bad2b80b1641a2de40cee65 WatchSource:0}: Error finding container aeb362ec1e41ee66286583ff027a94c24685c0644bad2b80b1641a2de40cee65: Status 404 returned error can't find the container with id aeb362ec1e41ee66286583ff027a94c24685c0644bad2b80b1641a2de40cee65 Sep 16 14:02:55.482756 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:55.482712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-de0b3e" event={"ID":"8bb8a784-0ad6-4da2-b204-ae5b50c07676","Type":"ContainerStarted","Data":"de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e"} Sep 16 14:02:55.482756 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:55.482757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-de0b3e" event={"ID":"8bb8a784-0ad6-4da2-b204-ae5b50c07676","Type":"ContainerStarted","Data":"aeb362ec1e41ee66286583ff027a94c24685c0644bad2b80b1641a2de40cee65"} Sep 16 14:02:55.501487 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:55.501429 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7896-de0b3e" podStartSLOduration=1.501414622 podStartE2EDuration="1.501414622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:55.499739835 +0000 UTC m=+1194.540333786" watchObservedRunningTime="2026-09-16 14:02:55.501414622 +0000 UTC m=+1194.542008612" Sep 16 14:02:55.687204 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:55.687164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:56.202266 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:56.202228 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-de0b3e"] Sep 16 14:02:56.207503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:56.207472 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-de0b3e"] Sep 16 14:02:56.488523 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:02:56.488494 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7896-de0b3e" container="test-curl-7896-de0b3e" Sep 16 14:02:56.689520 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:56.689488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:57.458031 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:57.457996 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:57.458511 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:57.458055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:57.473206 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:57.473178 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:57.494219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:57.494181 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7896-de0b3e" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" containerName="test-curl-7896-de0b3e" containerID="cri-o://de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e" gracePeriod=2 Sep 16 14:02:57.505346 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:57.505316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:02:59.658335 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:59.658305 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:02:59.782620 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:59.782525 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brj99\" (UniqueName: \"kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99\") pod \"8bb8a784-0ad6-4da2-b204-ae5b50c07676\" (UID: \"8bb8a784-0ad6-4da2-b204-ae5b50c07676\") " Sep 16 14:02:59.784642 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:59.784617 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99" (OuterVolumeSpecName: "kube-api-access-brj99") pod "8bb8a784-0ad6-4da2-b204-ae5b50c07676" (UID: "8bb8a784-0ad6-4da2-b204-ae5b50c07676"). InnerVolumeSpecName "kube-api-access-brj99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:59.883571 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:02:59.883538 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brj99\" (UniqueName: \"kubernetes.io/projected/8bb8a784-0ad6-4da2-b204-ae5b50c07676-kube-api-access-brj99\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:03:00.512329 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:00.512293 2559 generic.go:356] "Generic (PLEG): container finished" podID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" containerID="de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e" exitCode=137 Sep 16 14:03:00.512507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:00.512336 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:03:00.512507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:00.512389 2559 scope.go:117] "RemoveContainer" containerID="de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e" Sep 16 14:03:00.522639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:00.522619 2559 scope.go:117] "RemoveContainer" containerID="de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e" Sep 16 14:03:00.523071 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:00.523048 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e\": container with ID starting with de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e not found: ID does not exist" containerID="de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e" Sep 16 14:03:00.523071 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:00.523060 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-de0b3e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:03:00.523193 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:00.523079 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e"} err="failed to get container status \"de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e\": rpc error: code = NotFound desc = could not find container \"de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e\": container with ID starting with de8cf61bf52d0191ec94d61d3671cfa4f85f4680a36c6102bd93c83260900a6e not found: ID does not exist" Sep 16 14:03:01.321040 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.321001 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-f2a1be"] Sep 16 14:03:01.321421 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.321408 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" containerName="test-curl-7896-de0b3e" Sep 16 14:03:01.321480 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.321423 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" containerName="test-curl-7896-de0b3e" Sep 16 14:03:01.321516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.321507 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" containerName="test-curl-7896-de0b3e" Sep 16 14:03:01.325611 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.325592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:01.327718 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:01.327661 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-de0b3e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:03:01.327957 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.327939 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:03:01.331720 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.331692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-f2a1be"] Sep 16 14:03:01.397741 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.397708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmd2\" (UniqueName: \"kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2\") pod \"test-curl-7896-f2a1be\" (UID: \"4868c80c-f8f4-4d77-86d9-c5d42b09d136\") " pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:01.470275 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:01.470240 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-de0b3e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" pod="opendatahub/test-curl-7896-de0b3e" Sep 16 14:03:01.475781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.475747 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb8a784-0ad6-4da2-b204-ae5b50c07676" path="/var/lib/kubelet/pods/8bb8a784-0ad6-4da2-b204-ae5b50c07676/volumes" Sep 16 14:03:01.498310 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.498279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmd2\" (UniqueName: \"kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2\") pod \"test-curl-7896-f2a1be\" (UID: \"4868c80c-f8f4-4d77-86d9-c5d42b09d136\") " pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:01.508446 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.508417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmd2\" (UniqueName: \"kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2\") pod \"test-curl-7896-f2a1be\" (UID: \"4868c80c-f8f4-4d77-86d9-c5d42b09d136\") " pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:01.586150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.586061 2559 scope.go:117] "RemoveContainer" containerID="47674f840f8abdc390f1cb7b503d1fdc0691132838793c47441f0e7d77c6a9ef" Sep 16 14:03:01.637558 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.637530 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:03:01.645630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.645613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:01.924329 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:01.924303 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-f2a1be"] Sep 16 14:03:01.926419 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:03:01.926389 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4868c80c_f8f4_4d77_86d9_c5d42b09d136.slice/crio-05be15cfae4690217df78d138936be57072220137eed4332bf7d9bd555361188 WatchSource:0}: Error finding container 05be15cfae4690217df78d138936be57072220137eed4332bf7d9bd555361188: Status 404 returned error can't find the container with id 05be15cfae4690217df78d138936be57072220137eed4332bf7d9bd555361188 Sep 16 14:03:02.525833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:02.525782 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-f2a1be" event={"ID":"4868c80c-f8f4-4d77-86d9-c5d42b09d136","Type":"ContainerStarted","Data":"feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729"} Sep 16 14:03:02.525833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:02.525833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-f2a1be" event={"ID":"4868c80c-f8f4-4d77-86d9-c5d42b09d136","Type":"ContainerStarted","Data":"05be15cfae4690217df78d138936be57072220137eed4332bf7d9bd555361188"} Sep 16 14:03:02.542440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:02.542392 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7896-f2a1be" podStartSLOduration=1.542377957 podStartE2EDuration="1.542377957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:02.541202074 +0000 UTC m=+1201.581796026" watchObservedRunningTime="2026-09-16 14:03:02.542377957 +0000 UTC m=+1201.582971912" Sep 16 14:03:03.183454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:03.183413 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-f2a1be"] Sep 16 14:03:03.186538 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:03.186511 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-f2a1be"] Sep 16 14:03:03.530711 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:03.530684 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7896-f2a1be" container="test-curl-7896-f2a1be" Sep 16 14:03:04.535331 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:04.535286 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7896-f2a1be" podUID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" containerName="test-curl-7896-f2a1be" containerID="cri-o://feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729" gracePeriod=2 Sep 16 14:03:05.818968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:05.818925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:06.154749 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.154661 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:03:06.155115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.155101 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" containerName="test-curl-7896-f2a1be" Sep 16 14:03:06.155174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.155118 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" containerName="test-curl-7896-f2a1be" Sep 16 14:03:06.155210 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.155185 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" containerName="test-curl-7896-f2a1be" Sep 16 14:03:06.158405 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.158383 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.162094 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.162064 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 14:03:06.162207 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.162122 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ae7c3682-openshift-default-dockercfg-4ws79\"" Sep 16 14:03:06.180497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.180455 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:03:06.243457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243627 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243627 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243627 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmql\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243757 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.243786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.243774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344712 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmql\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.344927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.344899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345266 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.345796 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.345693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.347263 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.347240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.347808 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.347787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.359863 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.359822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmql\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.360145 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.360123 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:03:06.362626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.362595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token\") pod \"e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.468156 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.468118 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:06.681568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.681536 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:06.765758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.765731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:03:06.768044 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:03:06.768006 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c24a99_bb0c_42d6_8d71_ac7db71803dc.slice/crio-99fc1c4a133cc487f35e71c32af6c408f744764fd8b6417867517efbbf1ee59c WatchSource:0}: Error finding container 99fc1c4a133cc487f35e71c32af6c408f744764fd8b6417867517efbbf1ee59c: Status 404 returned error can't find the container with id 99fc1c4a133cc487f35e71c32af6c408f744764fd8b6417867517efbbf1ee59c Sep 16 14:03:06.774417 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.774368 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:06.774535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.774510 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:06.774622 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.774601 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:06.851211 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.851173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rmd2\" (UniqueName: \"kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2\") pod \"4868c80c-f8f4-4d77-86d9-c5d42b09d136\" (UID: \"4868c80c-f8f4-4d77-86d9-c5d42b09d136\") " Sep 16 14:03:06.853340 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.853307 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2" (OuterVolumeSpecName: "kube-api-access-7rmd2") pod "4868c80c-f8f4-4d77-86d9-c5d42b09d136" (UID: "4868c80c-f8f4-4d77-86d9-c5d42b09d136"). InnerVolumeSpecName "kube-api-access-7rmd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:06.953540 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:06.953490 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rmd2\" (UniqueName: \"kubernetes.io/projected/4868c80c-f8f4-4d77-86d9-c5d42b09d136-kube-api-access-7rmd2\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:03:07.144466 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.144381 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:07.262626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.262588 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:07.472753 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.472721 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" path="/var/lib/kubelet/pods/4868c80c-f8f4-4d77-86d9-c5d42b09d136/volumes" Sep 16 14:03:07.561887 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.561840 2559 generic.go:356] "Generic (PLEG): container finished" podID="4868c80c-f8f4-4d77-86d9-c5d42b09d136" containerID="feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729" exitCode=137 Sep 16 14:03:07.562073 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.561915 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-f2a1be" Sep 16 14:03:07.562073 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.561928 2559 scope.go:117] "RemoveContainer" containerID="feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729" Sep 16 14:03:07.564446 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.564409 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerStarted","Data":"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78"} Sep 16 14:03:07.564575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.564462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerStarted","Data":"99fc1c4a133cc487f35e71c32af6c408f744764fd8b6417867517efbbf1ee59c"} Sep 16 14:03:07.570978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.570956 2559 scope.go:117] "RemoveContainer" containerID="feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729" Sep 16 14:03:07.571259 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:03:07.571240 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729\": container with ID starting with feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729 not found: ID does not exist" containerID="feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729" Sep 16 14:03:07.571310 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.571268 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729"} err="failed to get container status \"feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729\": rpc error: code = NotFound desc = could not find container \"feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729\": container with ID starting with feb09048b43e7d4ac99f1c3e10510ddb5e27b69841f4ee7cf3e559ba97ee2729 not found: ID does not exist" Sep 16 14:03:07.587616 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:07.587573 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podStartSLOduration=1.587559763 podStartE2EDuration="1.587559763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:07.586109827 +0000 UTC m=+1206.626703781" watchObservedRunningTime="2026-09-16 14:03:07.587559763 +0000 UTC m=+1206.628153716" Sep 16 14:03:08.468713 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:08.468677 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:08.474175 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:08.474152 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:08.572144 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:08.572106 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:08.573007 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:08.572986 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:08.983671 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:08.983641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:09.769916 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:09.769853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:14.706768 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:14.706712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:15.131148 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:15.131008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:15.284012 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:15.283975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:15.525172 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:15.525140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:15.851603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:15.851507 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:15.947034 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:15.946987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:16.063410 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.063371 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:03:16.067830 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.067812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.070690 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.070666 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae7c3682\"" Sep 16 14:03:16.071253 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.071229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae7c3682-dockercfg-dl7m7\"" Sep 16 14:03:16.081150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.081126 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:03:16.145301 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.145218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.145301 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.145262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5c8\" (UniqueName: \"kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.246924 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.246848 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.247129 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.246947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5c8\" (UniqueName: \"kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.247576 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.247557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.257517 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.257482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5c8\" (UniqueName: \"kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8\") pod \"payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.377658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.377620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:16.398639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.398568 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:16.654533 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.654505 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:03:16.658851 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:03:16.658817 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-31657a30f45f8242719358fa65d357ebd21c07f6835092878761ad200633ae26 WatchSource:0}: Error finding container 31657a30f45f8242719358fa65d357ebd21c07f6835092878761ad200633ae26: Status 404 returned error can't find the container with id 31657a30f45f8242719358fa65d357ebd21c07f6835092878761ad200633ae26 Sep 16 14:03:16.892017 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:16.891973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:17.242394 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.242358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:17.553890 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.553784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:17.585283 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.585247 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:03:17.594922 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.594891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.597642 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.597613 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b4884e-openshift-default-dockercfg-hp4vd\"" Sep 16 14:03:17.609938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.609911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:03:17.622751 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.622710 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" event={"ID":"d02bf000-0ede-47f7-8bba-d16565e0ac10","Type":"ContainerStarted","Data":"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a"} Sep 16 14:03:17.622945 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.622768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" event={"ID":"d02bf000-0ede-47f7-8bba-d16565e0ac10","Type":"ContainerStarted","Data":"31657a30f45f8242719358fa65d357ebd21c07f6835092878761ad200633ae26"} Sep 16 14:03:17.623038 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.623019 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:17.651881 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.651813 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" podStartSLOduration=1.6517900490000001 podStartE2EDuration="1.651790049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:17.650047117 +0000 UTC m=+1216.690641071" watchObservedRunningTime="2026-09-16 14:03:17.651790049 +0000 UTC m=+1216.692384009" Sep 16 14:03:17.662133 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662095 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662285 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662160 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662285 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662212 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662285 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662316 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662369 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662490 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxngq\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.662490 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.662462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.667385 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.667346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:17.763250 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763250 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxngq\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763798 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763863 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.763863 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.763838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.764123 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.764104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.764223 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.764178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.764468 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.764443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.764565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.764509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.765968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.765942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.766921 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.766900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.775057 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.775021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.776353 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.776324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxngq\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq\") pod \"e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:17.905707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:17.905615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:18.000433 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.000401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:18.193551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.193524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:03:18.195449 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:03:18.195408 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83422dbd_ebe3_430c_9500_18b9af62f9dc.slice/crio-80a608044fae192bb5f6bd60827b8c7000fc7986a30c27122a273781ac5265ad WatchSource:0}: Error finding container 80a608044fae192bb5f6bd60827b8c7000fc7986a30c27122a273781ac5265ad: Status 404 returned error can't find the container with id 80a608044fae192bb5f6bd60827b8c7000fc7986a30c27122a273781ac5265ad Sep 16 14:03:18.198549 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.198515 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:18.198650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.198583 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:18.198650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.198610 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:18.584767 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.584732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:18.630897 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.630822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerStarted","Data":"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4"} Sep 16 14:03:18.631147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.630919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerStarted","Data":"80a608044fae192bb5f6bd60827b8c7000fc7986a30c27122a273781ac5265ad"} Sep 16 14:03:18.660856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.660790 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podStartSLOduration=1.6607669870000001 podStartE2EDuration="1.660766987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:18.659254755 +0000 UTC m=+1217.699848709" watchObservedRunningTime="2026-09-16 14:03:18.660766987 +0000 UTC m=+1217.701360946" Sep 16 14:03:18.906236 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:18.906153 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:19.707291 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:19.707259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:19.911725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:19.911685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:20.637500 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:20.637463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:20.641621 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:20.641595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:20.642604 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:20.642579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:21.631397 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:21.631366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:21.721337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:21.721302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:21.842000 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:21.841954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:22.990298 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:22.990261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:23.986555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:23.986518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.099255 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.099219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.458444 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.458406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.597261 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.597226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.762539 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.762453 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.867012 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.866981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:24.927178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.927145 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:03:24.931331 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.931305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:24.933894 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.933862 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b4884e\"" Sep 16 14:03:24.934013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.933961 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b4884e-dockercfg-fmmd6\"" Sep 16 14:03:24.941061 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:24.941034 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:03:25.043833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.043740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.043833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.043784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vhxc\" (UniqueName: \"kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.145064 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.145025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.145432 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.145072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vhxc\" (UniqueName: \"kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.145797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.145776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.155628 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.155598 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vhxc\" (UniqueName: \"kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc\") pod \"payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.220361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.220332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:25.243754 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.243726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.519137 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.519112 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:03:25.519834 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:03:25.519806 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102d71c7_0324_493d_bdc6_fc62c565c75a.slice/crio-0f1b336940254743015846bdad79c59eb08f853896fb7b6bfd22cc0c3cc2eea3 WatchSource:0}: Error finding container 0f1b336940254743015846bdad79c59eb08f853896fb7b6bfd22cc0c3cc2eea3: Status 404 returned error can't find the container with id 0f1b336940254743015846bdad79c59eb08f853896fb7b6bfd22cc0c3cc2eea3 Sep 16 14:03:25.677746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.677706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" event={"ID":"102d71c7-0324-493d-bdc6-fc62c565c75a","Type":"ContainerStarted","Data":"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439"} Sep 16 14:03:25.677952 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.677770 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" event={"ID":"102d71c7-0324-493d-bdc6-fc62c565c75a","Type":"ContainerStarted","Data":"0f1b336940254743015846bdad79c59eb08f853896fb7b6bfd22cc0c3cc2eea3"} Sep 16 14:03:25.677952 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.677812 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:25.698018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:25.697851 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" podStartSLOduration=1.6978322750000001 podStartE2EDuration="1.697832275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:25.697153194 +0000 UTC m=+1224.737747147" watchObservedRunningTime="2026-09-16 14:03:25.697832275 +0000 UTC m=+1224.738426230" Sep 16 14:03:26.108988 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:26.108952 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:26.997419 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:26.997370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:27.909559 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:27.909527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:28.632986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:28.632953 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:03:28.729374 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:28.729334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:29.751128 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:29.751096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:29.918426 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:29.918390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:30.712185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:30.712150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:31.894899 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:31.894847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:33.582064 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:33.582030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.727080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:35.727049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.826557 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:35.826521 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:36.081150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:36.081025 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:03:36.407628 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:36.407545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:36.685320 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:36.685235 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:03:37.587216 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:37.587175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:38.660662 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:38.660614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:39.481043 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:39.481008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:40.552714 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:40.552660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:46.249083 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.249041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:46.813373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.813279 2559 generic.go:356] "Generic (PLEG): container finished" podID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerID="3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78" exitCode=0 Sep 16 14:03:46.813373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.813359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerDied","Data":"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78"} Sep 16 14:03:46.813943 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.813922 2559 scope.go:117] "RemoveContainer" containerID="3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78" Sep 16 14:03:46.815177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815156 2559 generic.go:356] "Generic (PLEG): container finished" podID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerID="f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4" exitCode=0 Sep 16 14:03:46.815278 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerDied","Data":"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4"} Sep 16 14:03:46.815278 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815205 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:46.815278 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815273 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:46.815443 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815306 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:46.815520 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.815506 2559 scope.go:117] "RemoveContainer" containerID="f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4" Sep 16 14:03:46.816716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.816682 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:46.816804 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.816768 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:46.816884 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:46.816815 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:03:47.823015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:47.822976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerStarted","Data":"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765"} Sep 16 14:03:47.824955 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:47.824931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerStarted","Data":"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62"} Sep 16 14:03:47.906676 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:47.906645 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:47.907614 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:47.907587 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:47.907677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:47.907643 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:48.468978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:48.468930 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:48.469410 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:48.469379 2559 patch_prober.go:28] interesting pod/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 16 14:03:48.469492 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:48.469453 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 16 14:03:48.906862 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:48.906773 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:48.906862 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:48.906847 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:49.471312 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:49.471270 2559 patch_prober.go:28] interesting pod/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 16 14:03:49.471523 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:49.471339 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 16 14:03:49.905923 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:49.905842 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:49.905923 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:49.905916 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:50.469341 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:50.469297 2559 patch_prober.go:28] interesting pod/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 16 14:03:50.469821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:50.469364 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 16 14:03:50.642384 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:50.642348 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:50.906470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:50.906382 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:50.906470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:50.906438 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:51.470737 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:51.470702 2559 patch_prober.go:28] interesting pod/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 16 14:03:51.471355 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:51.471314 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 16 14:03:51.906154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:51.906061 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:51.906154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:51.906116 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:52.468984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:52.468942 2559 patch_prober.go:28] interesting pod/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 16 14:03:52.469217 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:52.469007 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 16 14:03:52.906527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:52.906437 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 16 14:03:52.906527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:52.906491 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 16 14:03:53.472968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:53.472937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:53.473196 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:53.473178 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:53.473999 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:53.473978 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:03:53.910437 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:53.910328 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:53.911214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:53.911195 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:03:56.335401 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:03:56.335364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.647917 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.647863 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:04:02.651612 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.651590 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.654231 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.654209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-19203d\"/\"distinct-2-w4-19203d-kserve-self-signed-certs\"" Sep 16 14:04:02.660186 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.660165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:04:02.745976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.745943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.746166 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.745981 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbzk\" (UniqueName: \"kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.746166 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.746091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.746166 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.746128 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.746327 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.746190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.746327 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.746239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.767433 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.767389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.847010 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.846973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847010 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847276 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847276 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847276 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847276 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbzk\" (UniqueName: \"kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847571 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847540 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.847616 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.847573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.849367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.849343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.849639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.849622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.856178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.856134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbzk\" (UniqueName: \"kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk\") pod \"distinct-2-w4-19203d-kserve-758c89dbc-mzv47\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:02.960862 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:02.960827 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:03.233156 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:03.233128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:04:03.235309 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:04:03.235285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod326b971e_f46a_4b38_8eca_c25fda559120.slice/crio-1af732921e7b55f39b6af6774c62b7a80a21f4d5e974c4b7ad9e59e2e9b9496d WatchSource:0}: Error finding container 1af732921e7b55f39b6af6774c62b7a80a21f4d5e974c4b7ad9e59e2e9b9496d: Status 404 returned error can't find the container with id 1af732921e7b55f39b6af6774c62b7a80a21f4d5e974c4b7ad9e59e2e9b9496d Sep 16 14:04:03.929547 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:03.929507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" event={"ID":"326b971e-f46a-4b38-8eca-c25fda559120","Type":"ContainerStarted","Data":"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a"} Sep 16 14:04:03.929547 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:03.929549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" event={"ID":"326b971e-f46a-4b38-8eca-c25fda559120","Type":"ContainerStarted","Data":"1af732921e7b55f39b6af6774c62b7a80a21f4d5e974c4b7ad9e59e2e9b9496d"} Sep 16 14:04:03.967275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:03.967215 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" podStartSLOduration=1.967200394 podStartE2EDuration="1.967200394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:04:03.966148677 +0000 UTC m=+1263.006742632" watchObservedRunningTime="2026-09-16 14:04:03.967200394 +0000 UTC m=+1263.007794361" Sep 16 14:04:06.552606 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:06.552563 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:04:12.961664 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:12.961609 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:12.962186 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:12.961691 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:12.973758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:12.973732 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:12.997290 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:12.997262 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:04:36.954644 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:36.954602 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:04:43.325216 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.325181 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:04:43.328113 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.328087 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.331176 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.331151 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"default-dockercfg-7k44r\"" Sep 16 14:04:43.331176 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.331168 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"kube-root-ca.crt\"" Sep 16 14:04:43.331377 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.331207 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"openshift-service-ca.crt\"" Sep 16 14:04:43.331377 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.331287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"auth-test-model-b4884e-kserve-self-signed-certs\"" Sep 16 14:04:43.337575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.337552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:04:43.462806 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.462772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ttc\" (UniqueName: \"kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.463009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.462830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.463009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.462911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.463009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.462985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.463154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.463021 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.463154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.463044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.563929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.563900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ttc\" (UniqueName: \"kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564062 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.563955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564062 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564062 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564204 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564204 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564473 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564441 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564473 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.564623 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.564566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.566177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.566155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.566307 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.566291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.573789 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.573767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ttc\" (UniqueName: \"kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc\") pod \"auth-test-model-b4884e-kserve-7b8fc7566c-l522c\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.639602 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.639547 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:43.915930 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:04:43.915893 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a96e849_dc7b_4c91_939d_8f2bdca87a73.slice/crio-a8345b0a4fa3b52977dd994b9faf6924c10bdf66a68803b62bb48fe3fa0ac148 WatchSource:0}: Error finding container a8345b0a4fa3b52977dd994b9faf6924c10bdf66a68803b62bb48fe3fa0ac148: Status 404 returned error can't find the container with id a8345b0a4fa3b52977dd994b9faf6924c10bdf66a68803b62bb48fe3fa0ac148 Sep 16 14:04:43.916267 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:43.916240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:04:44.133051 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:44.133003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:44.163355 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:44.163268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" event={"ID":"2a96e849-dc7b-4c91-939d-8f2bdca87a73","Type":"ContainerStarted","Data":"c4c1300348ba1f009bb471cb1e726b5d1d4cea39416c9d79c5ba602c4aa440a3"} Sep 16 14:04:44.163355 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:44.163317 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" event={"ID":"2a96e849-dc7b-4c91-939d-8f2bdca87a73","Type":"ContainerStarted","Data":"a8345b0a4fa3b52977dd994b9faf6924c10bdf66a68803b62bb48fe3fa0ac148"} Sep 16 14:04:44.185947 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:44.185902 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" podStartSLOduration=1.185887936 podStartE2EDuration="1.185887936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:04:44.183093726 +0000 UTC m=+1303.223687676" watchObservedRunningTime="2026-09-16 14:04:44.185887936 +0000 UTC m=+1303.226482273" Sep 16 14:04:45.861021 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:45.860989 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:46.808779 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:46.808741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:47.990369 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:47.990292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:48.805790 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:48.805753 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:49.723518 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:49.723483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:50.709392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:50.709355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:51.668352 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:51.668297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:52.556581 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:52.556542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:53.344997 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:53.344962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:53.639986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:53.639886 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:53.639986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:53.639934 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:53.652159 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:53.652130 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:54.231600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:54.231574 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:04:56.664522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:04:56.664488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:01.323486 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.323429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:01.391665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.391633 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:05:01.391898 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.391860 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" podUID="d02bf000-0ede-47f7-8bba-d16565e0ac10" containerName="payload-pre-processing" containerID="cri-o://9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a" gracePeriod=30 Sep 16 14:05:01.451522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.451488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:01.457096 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:01.457067 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-conmon-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:01.635793 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.635756 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:05:01.644565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.644541 2559 scope.go:117] "RemoveContainer" containerID="5f2138d1ceee3e1d5f99ad66ccbd5a5a5b98a648dc6c9177a5ed9eb9cdb886cf" Sep 16 14:05:01.754164 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.754126 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume\") pod \"d02bf000-0ede-47f7-8bba-d16565e0ac10\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " Sep 16 14:05:01.754366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.754216 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr5c8\" (UniqueName: \"kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8\") pod \"d02bf000-0ede-47f7-8bba-d16565e0ac10\" (UID: \"d02bf000-0ede-47f7-8bba-d16565e0ac10\") " Sep 16 14:05:01.754613 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.754583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d02bf000-0ede-47f7-8bba-d16565e0ac10" (UID: "d02bf000-0ede-47f7-8bba-d16565e0ac10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:01.756479 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.756442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8" (OuterVolumeSpecName: "kube-api-access-pr5c8") pod "d02bf000-0ede-47f7-8bba-d16565e0ac10" (UID: "d02bf000-0ede-47f7-8bba-d16565e0ac10"). InnerVolumeSpecName "kube-api-access-pr5c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:01.855373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.855329 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02bf000-0ede-47f7-8bba-d16565e0ac10-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:05:01.855373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:01.855364 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5c8\" (UniqueName: \"kubernetes.io/projected/d02bf000-0ede-47f7-8bba-d16565e0ac10-kube-api-access-pr5c8\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:05:02.005756 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.005725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:02.108666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.108628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:02.266799 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.266697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:02.272764 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.272727 2559 generic.go:356] "Generic (PLEG): container finished" podID="d02bf000-0ede-47f7-8bba-d16565e0ac10" containerID="9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a" exitCode=0 Sep 16 14:05:02.273021 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.272996 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" Sep 16 14:05:02.273164 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.272916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" event={"ID":"d02bf000-0ede-47f7-8bba-d16565e0ac10","Type":"ContainerDied","Data":"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a"} Sep 16 14:05:02.273225 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.273158 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr" event={"ID":"d02bf000-0ede-47f7-8bba-d16565e0ac10","Type":"ContainerDied","Data":"31657a30f45f8242719358fa65d357ebd21c07f6835092878761ad200633ae26"} Sep 16 14:05:02.273225 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.273194 2559 scope.go:117] "RemoveContainer" containerID="9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a" Sep 16 14:05:02.283280 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.283259 2559 scope.go:117] "RemoveContainer" containerID="9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a" Sep 16 14:05:02.283547 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:02.283531 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a\": container with ID starting with 9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a not found: ID does not exist" containerID="9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a" Sep 16 14:05:02.283596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.283556 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a"} err="failed to get container status \"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a\": rpc error: code = NotFound desc = could not find container \"9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a\": container with ID starting with 9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a not found: ID does not exist" Sep 16 14:05:02.303174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.303146 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:05:02.315485 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:02.315459 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae7c3682-5d774649db-n9lrr"] Sep 16 14:05:03.330344 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:03.330310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:03.472182 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:03.472146 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02bf000-0ede-47f7-8bba-d16565e0ac10" path="/var/lib/kubelet/pods/d02bf000-0ede-47f7-8bba-d16565e0ac10/volumes" Sep 16 14:05:04.235422 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:04.235393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:04.840512 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:04.840472 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:05.349813 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:05.349776 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:05.399562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:05.399526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:06.452730 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:06.452694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:07.698375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:07.698333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:07.803456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:07.803418 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.715994 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:08.715951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:09.919528 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:09.919490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:15.410491 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:15.410454 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:16.381958 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:16.381914 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:19.444660 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:19.444621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:21.955291 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:21.955250 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:05:23.115120 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:23.115082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:25.464851 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:25.464772 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:31.395264 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:31.395227 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:35.474783 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:35.474750 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:39.360897 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:39.360784 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:39.360897 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:39.360810 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:45.528985 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:45.528946 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:46.228834 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:46.228799 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:46.383640 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:46.383600 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:48.190503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:48.190459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:55.009139 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:55.009105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:55.590231 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:05:55.590197 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:05:59.462860 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:05:59.462821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:06:01.431479 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:06:01.431444 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02bf000_0ede_47f7_8bba_d16565e0ac10.slice/crio-9a0c69c5916410b3e8ff3d8281f32631d8b57137de2aa3aa9d82d3240319cc7a.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:06:01.604234 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.604199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:01.666146 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.666117 2559 scope.go:117] "RemoveContainer" containerID="d11fe8d98a11c551655fcf644fe0509811417bfb727a789e187861b2bb154a3f" Sep 16 14:06:01.675222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.675203 2559 scope.go:117] "RemoveContainer" containerID="4eb2b936404aa959bf4f0bda158a983990274be77f32ee911325e3b2b1e60c92" Sep 16 14:06:01.683301 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.683284 2559 scope.go:117] "RemoveContainer" containerID="8f0bd64037a9efcabee10cec89c7e2e2fc49eeb2d4746c793043afe6713f7a47" Sep 16 14:06:01.710825 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.710796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:01.940860 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.940773 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:06:01.941107 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.941021 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" containerID="cri-o://15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765" gracePeriod=30 Sep 16 14:06:01.941369 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:01.941344 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:03.440768 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:03.440731 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:03.547147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:03.547113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:04.161060 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:04.161009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:04.356880 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:04.356834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:05.608076 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:05.608043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:07.196603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.196577 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:06:07.322161 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322161 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322148 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322211 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322238 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322269 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvmql\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322358 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322633 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322389 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322633 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs\") pod \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\" (UID: \"08c24a99-bb0c-42d6-8d71-ac7db71803dc\") " Sep 16 14:06:07.322633 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322555 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:07.322839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322778 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:07.322930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322801 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:06:07.322930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322892 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:07.322930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322910 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:07.322930 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.322928 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.324531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.324490 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql" (OuterVolumeSpecName: "kube-api-access-lvmql") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "kube-api-access-lvmql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:07.324531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.324504 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:07.324531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.324492 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:07.324699 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.324590 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08c24a99-bb0c-42d6-8d71-ac7db71803dc" (UID: "08c24a99-bb0c-42d6-8d71-ac7db71803dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:06:07.423495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423458 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423489 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423497 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423505 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423514 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423521 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423529 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmql\" (UniqueName: \"kubernetes.io/projected/08c24a99-bb0c-42d6-8d71-ac7db71803dc-kube-api-access-lvmql\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.423781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.423538 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c24a99-bb0c-42d6-8d71-ac7db71803dc-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:07.640516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.640428 2559 generic.go:356] "Generic (PLEG): container finished" podID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerID="15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765" exitCode=0 Sep 16 14:06:07.640516 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.640481 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" Sep 16 14:06:07.640738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.640506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerDied","Data":"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765"} Sep 16 14:06:07.640738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.640561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s" event={"ID":"08c24a99-bb0c-42d6-8d71-ac7db71803dc","Type":"ContainerDied","Data":"99fc1c4a133cc487f35e71c32af6c408f744764fd8b6417867517efbbf1ee59c"} Sep 16 14:06:07.640738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.640585 2559 scope.go:117] "RemoveContainer" containerID="15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765" Sep 16 14:06:07.649798 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.649776 2559 scope.go:117] "RemoveContainer" containerID="3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78" Sep 16 14:06:07.658588 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.658536 2559 scope.go:117] "RemoveContainer" containerID="15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765" Sep 16 14:06:07.658921 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:06:07.658896 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765\": container with ID starting with 15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765 not found: ID does not exist" containerID="15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765" Sep 16 14:06:07.659010 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.658934 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765"} err="failed to get container status \"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765\": rpc error: code = NotFound desc = could not find container \"15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765\": container with ID starting with 15393924c22be12c2d3b19c4288b82d300dcb7e20cb6f4b31af7ac3d033d0765 not found: ID does not exist" Sep 16 14:06:07.659010 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.658958 2559 scope.go:117] "RemoveContainer" containerID="3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78" Sep 16 14:06:07.659259 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:06:07.659232 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78\": container with ID starting with 3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78 not found: ID does not exist" containerID="3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78" Sep 16 14:06:07.659404 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.659267 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78"} err="failed to get container status \"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78\": rpc error: code = NotFound desc = could not find container \"3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78\": container with ID starting with 3b94e60c2ca615feb1f282ff74f3274a42260bb314c18bf47d1c0318636c0a78 not found: ID does not exist" Sep 16 14:06:07.661138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.661118 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:06:07.665944 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:07.665922 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae7c3682-openshift-default-54d4577f48-dcr4s"] Sep 16 14:06:09.471816 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:09.471783 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" path="/var/lib/kubelet/pods/08c24a99-bb0c-42d6-8d71-ac7db71803dc/volumes" Sep 16 14:06:12.004666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:12.004634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:12.100851 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:12.100822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:12.677928 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:12.677893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:18.026438 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:18.026383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:19.433253 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:19.433220 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:21.291737 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:21.291697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:21.409457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:21.409422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:21.529655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:21.529605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:26.554773 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:26.554738 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:26.654262 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:26.654213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:26.827819 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:26.827713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:27.595290 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:27.595255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:28.411698 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:28.411655 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:28.528583 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:28.528544 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:06:28.558290 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:28.558256 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 14:06:28.581367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:28.581338 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 14:06:28.648321 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:28.648290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:30.386862 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:30.386824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:30.488002 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:30.487967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:34.916587 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:34.916552 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:35.021371 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:35.021333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:36.046747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:36.046711 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:06:39.378161 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.378077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:39.386596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.386569 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r"] Sep 16 14:06:39.391428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.391402 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035c64-29826120-h2f5r"] Sep 16 14:06:39.431839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.431804 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 14:06:39.432080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.432054 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" podUID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" containerName="payload-pre-processing" containerID="cri-o://26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122" gracePeriod=30 Sep 16 14:06:39.473149 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.473118 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8895fcb-d369-4064-aba7-0b20dd2d4e19" path="/var/lib/kubelet/pods/d8895fcb-d369-4064-aba7-0b20dd2d4e19/volumes" Sep 16 14:06:39.648721 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.648602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:39.684508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.684483 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 14:06:39.738627 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.738597 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume\") pod \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " Sep 16 14:06:39.738801 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.738659 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwrwf\" (UniqueName: \"kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf\") pod \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\" (UID: \"1d306d8c-0c25-4f19-9b8c-e425d3fd4634\") " Sep 16 14:06:39.739082 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.739056 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d306d8c-0c25-4f19-9b8c-e425d3fd4634" (UID: "1d306d8c-0c25-4f19-9b8c-e425d3fd4634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:06:39.740729 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.740698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf" (OuterVolumeSpecName: "kube-api-access-cwrwf") pod "1d306d8c-0c25-4f19-9b8c-e425d3fd4634" (UID: "1d306d8c-0c25-4f19-9b8c-e425d3fd4634"). InnerVolumeSpecName "kube-api-access-cwrwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:39.769130 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.769100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:39.832416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.832381 2559 generic.go:356] "Generic (PLEG): container finished" podID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" containerID="26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122" exitCode=0 Sep 16 14:06:39.832595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.832427 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" Sep 16 14:06:39.832595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.832467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" event={"ID":"1d306d8c-0c25-4f19-9b8c-e425d3fd4634","Type":"ContainerDied","Data":"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122"} Sep 16 14:06:39.832595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.832518 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42" event={"ID":"1d306d8c-0c25-4f19-9b8c-e425d3fd4634","Type":"ContainerDied","Data":"d125c0ddb6381b64b62d5ac68eb5772891f92017a924b68ae96fc91b4dee2b45"} Sep 16 14:06:39.832595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.832541 2559 scope.go:117] "RemoveContainer" containerID="26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122" Sep 16 14:06:39.840296 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.840266 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrwf\" (UniqueName: \"kubernetes.io/projected/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-kube-api-access-cwrwf\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.840465 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.840443 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d306d8c-0c25-4f19-9b8c-e425d3fd4634-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.842030 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.842013 2559 scope.go:117] "RemoveContainer" containerID="26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122" Sep 16 14:06:39.842326 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:06:39.842310 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122\": container with ID starting with 26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122 not found: ID does not exist" containerID="26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122" Sep 16 14:06:39.842377 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.842334 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122"} err="failed to get container status \"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122\": rpc error: code = NotFound desc = could not find container \"26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122\": container with ID starting with 26d805381f31ddf1fc4bbd4ee87750a2931eb55fc48c788f20f2330320ba0122 not found: ID does not exist" Sep 16 14:06:39.871745 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.871701 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 14:06:39.886228 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:39.886191 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035c64-796d578f4-csw42"] Sep 16 14:06:40.169035 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:40.168996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:40.293513 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:40.293480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:41.472374 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:41.472345 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" path="/var/lib/kubelet/pods/1d306d8c-0c25-4f19-9b8c-e425d3fd4634/volumes" Sep 16 14:06:41.548180 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:41.548143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:42.738049 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:42.738018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:43.547724 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.547657 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" podUID="35135d14-f3f3-4a3a-8562-bd3f04e38595" containerName="main" containerID="cri-o://2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e" gracePeriod=45 Sep 16 14:06:43.576477 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.576417 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" podUID="33e42127-44de-442a-bfc9-47b4135510ef" containerName="main" containerID="cri-o://cd9c5cfb94b125040a77092357ee8f9ef9324ee8217c0cb14e0a23487f333af5" gracePeriod=45 Sep 16 14:06:43.599411 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.599376 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" podUID="0314deaf-0aed-447e-b279-222e6919dd81" containerName="main" containerID="cri-o://6cf7abfd01e8235829e0c383c7f2a45f75eb32789e159a851283706432f7ceac" gracePeriod=45 Sep 16 14:06:43.849568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.849538 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:06:43.856968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.856931 2559 generic.go:356] "Generic (PLEG): container finished" podID="33e42127-44de-442a-bfc9-47b4135510ef" containerID="cd9c5cfb94b125040a77092357ee8f9ef9324ee8217c0cb14e0a23487f333af5" exitCode=0 Sep 16 14:06:43.857135 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.857000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" event={"ID":"33e42127-44de-442a-bfc9-47b4135510ef","Type":"ContainerDied","Data":"cd9c5cfb94b125040a77092357ee8f9ef9324ee8217c0cb14e0a23487f333af5"} Sep 16 14:06:43.859004 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.858979 2559 generic.go:356] "Generic (PLEG): container finished" podID="0314deaf-0aed-447e-b279-222e6919dd81" containerID="6cf7abfd01e8235829e0c383c7f2a45f75eb32789e159a851283706432f7ceac" exitCode=0 Sep 16 14:06:43.859138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.859062 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" event={"ID":"0314deaf-0aed-447e-b279-222e6919dd81","Type":"ContainerDied","Data":"6cf7abfd01e8235829e0c383c7f2a45f75eb32789e159a851283706432f7ceac"} Sep 16 14:06:43.860661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.860637 2559 generic.go:356] "Generic (PLEG): container finished" podID="35135d14-f3f3-4a3a-8562-bd3f04e38595" containerID="2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e" exitCode=0 Sep 16 14:06:43.860778 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.860676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" event={"ID":"35135d14-f3f3-4a3a-8562-bd3f04e38595","Type":"ContainerDied","Data":"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e"} Sep 16 14:06:43.860778 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.860704 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" event={"ID":"35135d14-f3f3-4a3a-8562-bd3f04e38595","Type":"ContainerDied","Data":"2576cb1fb2c24d18981d23ba9e9aac2e5e0ad1c30e3684e520fd0ead4a4a15b8"} Sep 16 14:06:43.860778 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.860677 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659" Sep 16 14:06:43.860778 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.860726 2559 scope.go:117] "RemoveContainer" containerID="2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e" Sep 16 14:06:43.886517 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.885943 2559 scope.go:117] "RemoveContainer" containerID="2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e" Sep 16 14:06:43.886608 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886564 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.886608 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886602 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.886722 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886653 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.886722 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886702 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kddh\" (UniqueName: \"kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.886837 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886734 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.886920 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.886899 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache\") pod \"35135d14-f3f3-4a3a-8562-bd3f04e38595\" (UID: \"35135d14-f3f3-4a3a-8562-bd3f04e38595\") " Sep 16 14:06:43.888458 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:06:43.888388 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e\": container with ID starting with 2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e not found: ID does not exist" containerID="2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e" Sep 16 14:06:43.888458 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.888425 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e"} err="failed to get container status \"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e\": rpc error: code = NotFound desc = could not find container \"2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e\": container with ID starting with 2edee3e884061d1e27bfb54b7599ce675eb549c8c81204d2a1880488476f519e not found: ID does not exist" Sep 16 14:06:43.888689 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.888468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.888936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.888900 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home" (OuterVolumeSpecName: "home") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.889740 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.889715 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache" (OuterVolumeSpecName: "model-cache") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.890185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.890154 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm" (OuterVolumeSpecName: "dshm") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.890276 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.890226 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:06:43.891391 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.891370 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh" (OuterVolumeSpecName: "kube-api-access-9kddh") pod "35135d14-f3f3-4a3a-8562-bd3f04e38595" (UID: "35135d14-f3f3-4a3a-8562-bd3f04e38595"). InnerVolumeSpecName "kube-api-access-9kddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:43.892777 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.892760 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 14:06:43.911203 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.911174 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 14:06:43.987747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.987713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.987997 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.987773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.987997 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.987820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.988115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988029 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rcml\" (UniqueName: \"kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.988115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988031 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home" (OuterVolumeSpecName: "home") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.988115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.988115 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988100 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988131 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988150 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988180 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j696\" (UniqueName: \"kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988210 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988243 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988278 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir\") pod \"0314deaf-0aed-447e-b279-222e6919dd81\" (UID: \"0314deaf-0aed-447e-b279-222e6919dd81\") " Sep 16 14:06:43.988319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988306 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs\") pod \"33e42127-44de-442a-bfc9-47b4135510ef\" (UID: \"33e42127-44de-442a-bfc9-47b4135510ef\") " Sep 16 14:06:43.988658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988640 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988658 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988672 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988686 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35135d14-f3f3-4a3a-8562-bd3f04e38595-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988699 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kddh\" (UniqueName: \"kubernetes.io/projected/35135d14-f3f3-4a3a-8562-bd3f04e38595-kube-api-access-9kddh\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988716 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988712 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35135d14-f3f3-4a3a-8562-bd3f04e38595-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988725 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988736 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988304 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home" (OuterVolumeSpecName: "home") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988408 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache" (OuterVolumeSpecName: "model-cache") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.988985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.988577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.990535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.990510 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm" (OuterVolumeSpecName: "dshm") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:43.990535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.990527 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml" (OuterVolumeSpecName: "kube-api-access-9rcml") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "kube-api-access-9rcml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:43.990710 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.990664 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696" (OuterVolumeSpecName: "kube-api-access-4j696") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "kube-api-access-4j696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:43.991225 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.991207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:06:43.991357 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.991334 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e42127-44de-442a-bfc9-47b4135510ef" (UID: "33e42127-44de-442a-bfc9-47b4135510ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:06:43.991694 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:43.991670 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm" (OuterVolumeSpecName: "dshm") pod "0314deaf-0aed-447e-b279-222e6919dd81" (UID: "0314deaf-0aed-447e-b279-222e6919dd81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:44.026173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.026131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089242 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089273 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089282 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e42127-44de-442a-bfc9-47b4135510ef-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089290 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089299 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rcml\" (UniqueName: \"kubernetes.io/projected/33e42127-44de-442a-bfc9-47b4135510ef-kube-api-access-9rcml\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089307 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089315 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0314deaf-0aed-447e-b279-222e6919dd81-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089323 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0314deaf-0aed-447e-b279-222e6919dd81-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089331 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j696\" (UniqueName: \"kubernetes.io/projected/0314deaf-0aed-447e-b279-222e6919dd81-kube-api-access-4j696\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.089723 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.089338 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e42127-44de-442a-bfc9-47b4135510ef-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:06:44.185308 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.185271 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:06:44.200129 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.200094 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-w0-035c64-kserve-59b6475ddd-9t659"] Sep 16 14:06:44.868630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.868546 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" Sep 16 14:06:44.868630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.868577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr" event={"ID":"0314deaf-0aed-447e-b279-222e6919dd81","Type":"ContainerDied","Data":"5dfb3227c2675a5e6600eb13ed85968eb0f15f5c3692c7560ba2051add0860ea"} Sep 16 14:06:44.868630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.868626 2559 scope.go:117] "RemoveContainer" containerID="6cf7abfd01e8235829e0c383c7f2a45f75eb32789e159a851283706432f7ceac" Sep 16 14:06:44.871159 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.871132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" event={"ID":"33e42127-44de-442a-bfc9-47b4135510ef","Type":"ContainerDied","Data":"db0cac2b1ffafb3cc6e12628f0c20d239431531a8f26557c7492b1b0666820a9"} Sep 16 14:06:44.871274 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.871146 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc" Sep 16 14:06:44.877763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.877746 2559 scope.go:117] "RemoveContainer" containerID="cd9c5cfb94b125040a77092357ee8f9ef9324ee8217c0cb14e0a23487f333af5" Sep 16 14:06:44.893610 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.893582 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 14:06:44.893953 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.893930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:44.898234 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.898213 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/simulator-w0-035c64-kserve-758fc5cc9b-rg4zr"] Sep 16 14:06:44.912361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.912335 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 14:06:44.918449 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:44.918421 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/premium-w0-035c64-kserve-65f4d4f48f-gm7xc"] Sep 16 14:06:45.473296 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:45.473259 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0314deaf-0aed-447e-b279-222e6919dd81" path="/var/lib/kubelet/pods/0314deaf-0aed-447e-b279-222e6919dd81/volumes" Sep 16 14:06:45.473648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:45.473635 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e42127-44de-442a-bfc9-47b4135510ef" path="/var/lib/kubelet/pods/33e42127-44de-442a-bfc9-47b4135510ef/volumes" Sep 16 14:06:45.474003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:45.473991 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35135d14-f3f3-4a3a-8562-bd3f04e38595" path="/var/lib/kubelet/pods/35135d14-f3f3-4a3a-8562-bd3f04e38595/volumes" Sep 16 14:06:45.892922 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:45.892820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:47.362879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:47.362833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:47.507061 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:47.507027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:47.748147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:47.748109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:53.235184 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:06:53.235150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:01.179051 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.179011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:01.324047 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.324011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:01.476839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.476806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:01.717090 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.717057 2559 scope.go:117] "RemoveContainer" containerID="47d4d039075654e51ea1439efe060d29a6318e29b2fc89377807ce2cd0ec3c49" Sep 16 14:07:01.725523 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.725502 2559 scope.go:117] "RemoveContainer" containerID="fecf7d4cfbc364263832c6b15071d0c508663b8c3452e43cc8bf0e176200b5cc" Sep 16 14:07:01.900334 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:01.900300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:02.137197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:02.137109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:04.763806 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:04.763775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:05.428758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:05.428725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:05.538151 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:05.538116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:06.186941 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:06.186898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:06.626279 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:06.626245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:07.449588 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:07.449543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:07.995442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:07.995412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:08.596313 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:08.596276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:08.870167 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:08.870053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:09.878897 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:09.878819 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:09.982082 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:09.982045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:10.266303 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:10.266263 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:07:11.008976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:11.008943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:11.146709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:11.146674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:12.247111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:12.247072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:12.360096 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:12.360056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:13.041522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.041474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:13.219723 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.219684 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:07:13.220093 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220081 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220093 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220095 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220107 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220115 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220130 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0314deaf-0aed-447e-b279-222e6919dd81" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220136 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0314deaf-0aed-447e-b279-222e6919dd81" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220146 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" containerName="payload-pre-processing" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220151 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" containerName="payload-pre-processing" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220160 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35135d14-f3f3-4a3a-8562-bd3f04e38595" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220165 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35135d14-f3f3-4a3a-8562-bd3f04e38595" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220172 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e42127-44de-442a-bfc9-47b4135510ef" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220176 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e42127-44de-442a-bfc9-47b4135510ef" containerName="main" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220185 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02bf000-0ede-47f7-8bba-d16565e0ac10" containerName="payload-pre-processing" Sep 16 14:07:13.220185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220190 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02bf000-0ede-47f7-8bba-d16565e0ac10" containerName="payload-pre-processing" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220247 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02bf000-0ede-47f7-8bba-d16565e0ac10" containerName="payload-pre-processing" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220254 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e42127-44de-442a-bfc9-47b4135510ef" containerName="main" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220261 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d306d8c-0c25-4f19-9b8c-e425d3fd4634" containerName="payload-pre-processing" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220267 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220274 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35135d14-f3f3-4a3a-8562-bd3f04e38595" containerName="main" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220280 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c24a99-bb0c-42d6-8d71-ac7db71803dc" containerName="istio-proxy" Sep 16 14:07:13.220570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.220286 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0314deaf-0aed-447e-b279-222e6919dd81" containerName="main" Sep 16 14:07:13.224974 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.224956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.226984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.226956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:13.227938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.227921 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7f75a3\"" Sep 16 14:07:13.239856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.239831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:07:13.270303 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.270271 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:07:13.273672 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.273656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.275659 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.275638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.275759 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.275670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bctg\" (UniqueName: \"kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.275759 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.275724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.275890 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.275850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdx4\" (UniqueName: \"kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.298475 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.298405 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7f75a3-dockercfg-9v2jb\"" Sep 16 14:07:13.316209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.316178 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:07:13.376799 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.376759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdx4\" (UniqueName: \"kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.377003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.376810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.377003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.376829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bctg\" (UniqueName: \"kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.377003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.376912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.377483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.377460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.377626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.377606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.392050 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.392021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdx4\" (UniqueName: \"kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4\") pod \"payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.394068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.394044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bctg\" (UniqueName: \"kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg\") pod \"payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.535491 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.535452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:13.584245 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.584217 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:13.821179 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.821149 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:07:13.823049 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:07:13.823017 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5d3caa_cdfe_455c_978b_e80ca972daa6.slice/crio-12ac43b3a6c75367c006fd5157998a7a111ebc476fc3b95d69d42f86bc9ccaaa WatchSource:0}: Error finding container 12ac43b3a6c75367c006fd5157998a7a111ebc476fc3b95d69d42f86bc9ccaaa: Status 404 returned error can't find the container with id 12ac43b3a6c75367c006fd5157998a7a111ebc476fc3b95d69d42f86bc9ccaaa Sep 16 14:07:13.876208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:13.876185 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:07:13.877103 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:07:13.877079 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf641f6_b35c_4f10_8513_e0668791b63b.slice/crio-8cc29d25a18677affdd7725e69941cf19b104a080bfdeefdaa4a2688674198a7 WatchSource:0}: Error finding container 8cc29d25a18677affdd7725e69941cf19b104a080bfdeefdaa4a2688674198a7: Status 404 returned error can't find the container with id 8cc29d25a18677affdd7725e69941cf19b104a080bfdeefdaa4a2688674198a7 Sep 16 14:07:14.029202 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.029163 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" event={"ID":"6b5d3caa-cdfe-455c-978b-e80ca972daa6","Type":"ContainerStarted","Data":"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0"} Sep 16 14:07:14.029397 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.029216 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" event={"ID":"6b5d3caa-cdfe-455c-978b-e80ca972daa6","Type":"ContainerStarted","Data":"12ac43b3a6c75367c006fd5157998a7a111ebc476fc3b95d69d42f86bc9ccaaa"} Sep 16 14:07:14.029397 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.029245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:14.030851 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.030824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" event={"ID":"adf641f6-b35c-4f10-8513-e0668791b63b","Type":"ContainerStarted","Data":"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4"} Sep 16 14:07:14.030992 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.030885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" event={"ID":"adf641f6-b35c-4f10-8513-e0668791b63b","Type":"ContainerStarted","Data":"8cc29d25a18677affdd7725e69941cf19b104a080bfdeefdaa4a2688674198a7"} Sep 16 14:07:14.030992 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.030908 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:14.070413 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:14.070335 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" podStartSLOduration=1.070281473 podStartE2EDuration="1.070281473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:14.068311189 +0000 UTC m=+1453.108905141" watchObservedRunningTime="2026-09-16 14:07:14.070281473 +0000 UTC m=+1453.110875427" Sep 16 14:07:17.483844 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:17.483802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:22.128213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:22.128178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:24.125519 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.125483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:24.275551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.275507 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:24.362766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.362715 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" podStartSLOduration=11.362698495 podStartE2EDuration="11.362698495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:14.105332514 +0000 UTC m=+1453.145926468" watchObservedRunningTime="2026-09-16 14:07:24.362698495 +0000 UTC m=+1463.403292448" Sep 16 14:07:24.364689 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.364666 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:07:24.374981 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.374952 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:07:24.386836 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.386751 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 14:07:24.408705 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.408671 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 14:07:24.528440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.528406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:24.767361 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:24.767327 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:25.038347 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:25.038254 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:07:25.038520 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:25.038510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:07:25.778491 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:25.778438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:26.987201 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:26.987162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:28.080447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:28.080411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:29.098358 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:29.098312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:30.096849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:30.096809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:30.280009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:30.279972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:30.761142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:30.761090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:31.085079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:31.084980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:31.301333 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:31.301271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:31.424637 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:31.424546 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:32.053912 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:32.053856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:07:32.978079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:32.978041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:33.201587 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:33.201545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:34.407617 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:34.407573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:35.312807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:35.312769 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:35.439545 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:35.439513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:35.848322 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:35.848286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:37.477938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:37.477906 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:39.071230 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.071187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:39.383756 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.383642 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" podUID="326b971e-f46a-4b38-8eca-c25fda559120" containerName="main" containerID="cri-o://eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a" gracePeriod=45 Sep 16 14:07:39.394551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.394511 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" podUID="09aae84e-476b-410d-98f9-86f02fc5823b" containerName="main" containerID="cri-o://90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400" gracePeriod=45 Sep 16 14:07:39.405572 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.405516 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" podUID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" containerName="main" containerID="cri-o://e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586" gracePeriod=45 Sep 16 14:07:39.426805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.426762 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" podUID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" containerName="main" containerID="cri-o://e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702" gracePeriod=45 Sep 16 14:07:39.696513 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.696485 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:07:39.719049 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.719026 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:07:39.725518 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.725498 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 14:07:39.737526 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.737500 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 14:07:39.831981 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.831948 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832038 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832063 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wts7b\" (UniqueName: \"kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832126 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.832168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832154 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832187 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832233 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832260 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home" (OuterVolumeSpecName: "home") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832268 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q5z8\" (UniqueName: \"kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832302 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832368 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832415 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.832497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832498 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832505 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832583 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache" (OuterVolumeSpecName: "model-cache") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832606 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832624 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832656 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm\") pod \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\" (UID: \"3a82cfad-f3c8-4241-bfb4-137d4d8e5218\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832696 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrcfg\" (UniqueName: \"kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg\") pod \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\" (UID: \"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs\") pod \"09aae84e-476b-410d-98f9-86f02fc5823b\" (UID: \"09aae84e-476b-410d-98f9-86f02fc5823b\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832792 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffbzk\" (UniqueName: \"kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk\") pod \"326b971e-f46a-4b38-8eca-c25fda559120\" (UID: \"326b971e-f46a-4b38-8eca-c25fda559120\") " Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.832881 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833068 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833031 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache" (OuterVolumeSpecName: "model-cache") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833211 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833230 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833243 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833256 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833268 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833281 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833293 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.833766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.833305 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.834489 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.834459 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home" (OuterVolumeSpecName: "home") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.835433 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.835406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home" (OuterVolumeSpecName: "home") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.835539 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.835442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8" (OuterVolumeSpecName: "kube-api-access-4q5z8") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "kube-api-access-4q5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:07:39.835539 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.835471 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:39.835665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.835647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home" (OuterVolumeSpecName: "home") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.836527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.836478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.837503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.837454 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:39.838032 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.837974 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm" (OuterVolumeSpecName: "dshm") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.838032 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.837982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:39.838208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838083 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm" (OuterVolumeSpecName: "dshm") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.838208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838141 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm" (OuterVolumeSpecName: "dshm") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.838208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838181 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk" (OuterVolumeSpecName: "kube-api-access-ffbzk") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "kube-api-access-ffbzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:07:39.838208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838192 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg" (OuterVolumeSpecName: "kube-api-access-wrcfg") pod "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" (UID: "a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff"). InnerVolumeSpecName "kube-api-access-wrcfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:07:39.838639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838609 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b" (OuterVolumeSpecName: "kube-api-access-wts7b") pod "09aae84e-476b-410d-98f9-86f02fc5823b" (UID: "09aae84e-476b-410d-98f9-86f02fc5823b"). InnerVolumeSpecName "kube-api-access-wts7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:07:39.838963 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.838925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "326b971e-f46a-4b38-8eca-c25fda559120" (UID: "326b971e-f46a-4b38-8eca-c25fda559120"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:39.839409 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.839389 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm" (OuterVolumeSpecName: "dshm") pod "3a82cfad-f3c8-4241-bfb4-137d4d8e5218" (UID: "3a82cfad-f3c8-4241-bfb4-137d4d8e5218"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934438 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09aae84e-476b-410d-98f9-86f02fc5823b-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934469 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbzk\" (UniqueName: \"kubernetes.io/projected/326b971e-f46a-4b38-8eca-c25fda559120-kube-api-access-ffbzk\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934478 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934486 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wts7b\" (UniqueName: \"kubernetes.io/projected/09aae84e-476b-410d-98f9-86f02fc5823b-kube-api-access-wts7b\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934495 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q5z8\" (UniqueName: \"kubernetes.io/projected/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-kube-api-access-4q5z8\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934505 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934514 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326b971e-f46a-4b38-8eca-c25fda559120-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934521 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934529 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326b971e-f46a-4b38-8eca-c25fda559120-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934536 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09aae84e-476b-410d-98f9-86f02fc5823b-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934543 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934550 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934557 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934564 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a82cfad-f3c8-4241-bfb4-137d4d8e5218-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934571 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:39.934978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:39.934578 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrcfg\" (UniqueName: \"kubernetes.io/projected/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff-kube-api-access-wrcfg\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:07:40.174066 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.174029 2559 generic.go:356] "Generic (PLEG): container finished" podID="09aae84e-476b-410d-98f9-86f02fc5823b" containerID="90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400" exitCode=0 Sep 16 14:07:40.174488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.174069 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" Sep 16 14:07:40.174488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.174112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" event={"ID":"09aae84e-476b-410d-98f9-86f02fc5823b","Type":"ContainerDied","Data":"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400"} Sep 16 14:07:40.174488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.174161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp" event={"ID":"09aae84e-476b-410d-98f9-86f02fc5823b","Type":"ContainerDied","Data":"a9a9da488659250afd7462a5643f50552b6f040c9076bfdd303a98f9ade535d5"} Sep 16 14:07:40.174488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.174181 2559 scope.go:117] "RemoveContainer" containerID="90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400" Sep 16 14:07:40.175469 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.175447 2559 generic.go:356] "Generic (PLEG): container finished" podID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" containerID="e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586" exitCode=0 Sep 16 14:07:40.175556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.175487 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" Sep 16 14:07:40.175556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.175528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" event={"ID":"3a82cfad-f3c8-4241-bfb4-137d4d8e5218","Type":"ContainerDied","Data":"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586"} Sep 16 14:07:40.175661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.175563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s" event={"ID":"3a82cfad-f3c8-4241-bfb4-137d4d8e5218","Type":"ContainerDied","Data":"71c7d4d757c9501948344a08d7d62dd7ac8efb32fa9da7aacfcbef6d803fff28"} Sep 16 14:07:40.177083 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.177065 2559 generic.go:356] "Generic (PLEG): container finished" podID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" containerID="e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702" exitCode=0 Sep 16 14:07:40.177187 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.177092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" event={"ID":"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff","Type":"ContainerDied","Data":"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702"} Sep 16 14:07:40.177187 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.177110 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" Sep 16 14:07:40.177275 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.177208 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs" event={"ID":"a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff","Type":"ContainerDied","Data":"7296fe8dc18be42c38180fce161d6e16c6d2aea88c200831a00736181852c803"} Sep 16 14:07:40.178638 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.178615 2559 generic.go:356] "Generic (PLEG): container finished" podID="326b971e-f46a-4b38-8eca-c25fda559120" containerID="eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a" exitCode=0 Sep 16 14:07:40.178728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.178657 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" Sep 16 14:07:40.178728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.178697 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" event={"ID":"326b971e-f46a-4b38-8eca-c25fda559120","Type":"ContainerDied","Data":"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a"} Sep 16 14:07:40.178828 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.178733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47" event={"ID":"326b971e-f46a-4b38-8eca-c25fda559120","Type":"ContainerDied","Data":"1af732921e7b55f39b6af6774c62b7a80a21f4d5e974c4b7ad9e59e2e9b9496d"} Sep 16 14:07:40.184389 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.184370 2559 scope.go:117] "RemoveContainer" containerID="90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400" Sep 16 14:07:40.184743 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:07:40.184689 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400\": container with ID starting with 90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400 not found: ID does not exist" containerID="90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400" Sep 16 14:07:40.184743 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.184723 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400"} err="failed to get container status \"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400\": rpc error: code = NotFound desc = could not find container \"90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400\": container with ID starting with 90bac4994073563551b7e641665c78997f394b40c6de394b6c140206d0741400 not found: ID does not exist" Sep 16 14:07:40.184915 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.184744 2559 scope.go:117] "RemoveContainer" containerID="e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586" Sep 16 14:07:40.194602 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.194588 2559 scope.go:117] "RemoveContainer" containerID="e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586" Sep 16 14:07:40.194891 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:07:40.194851 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586\": container with ID starting with e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586 not found: ID does not exist" containerID="e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586" Sep 16 14:07:40.194960 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.194891 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586"} err="failed to get container status \"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586\": rpc error: code = NotFound desc = could not find container \"e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586\": container with ID starting with e5ea414476e71979293f1f02a8790c1590938a59d6417d445421759a33065586 not found: ID does not exist" Sep 16 14:07:40.194960 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.194907 2559 scope.go:117] "RemoveContainer" containerID="e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702" Sep 16 14:07:40.202211 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.202194 2559 scope.go:117] "RemoveContainer" containerID="e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702" Sep 16 14:07:40.202442 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:07:40.202423 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702\": container with ID starting with e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702 not found: ID does not exist" containerID="e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702" Sep 16 14:07:40.202487 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.202448 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702"} err="failed to get container status \"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702\": rpc error: code = NotFound desc = could not find container \"e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702\": container with ID starting with e780e62d76f4df0caa416c4d752df11aa98975310dec089409b1105738c0b702 not found: ID does not exist" Sep 16 14:07:40.202487 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.202464 2559 scope.go:117] "RemoveContainer" containerID="eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a" Sep 16 14:07:40.210593 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.210574 2559 scope.go:117] "RemoveContainer" containerID="eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a" Sep 16 14:07:40.210864 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:07:40.210845 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a\": container with ID starting with eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a not found: ID does not exist" containerID="eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a" Sep 16 14:07:40.210963 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.210881 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a"} err="failed to get container status \"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a\": rpc error: code = NotFound desc = could not find container \"eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a\": container with ID starting with eb62f73a70d6380afa4ad8b70274e7baa2fedf9f273ce178cb56609c2798ca4a not found: ID does not exist" Sep 16 14:07:40.232805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.232778 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:07:40.238436 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.238408 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-w4-19203d-kserve-97d9d56d-78hqp"] Sep 16 14:07:40.265403 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.265378 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 14:07:40.274623 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.274601 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/simulator-w4-19203d-kserve-7cc98d9996-x4zqs"] Sep 16 14:07:40.298800 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.298778 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:07:40.302043 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.302023 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/distinct-2-w4-19203d-kserve-758c89dbc-mzv47"] Sep 16 14:07:40.319507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.319485 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 14:07:40.330222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.330200 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-19203d/premium-w4-19203d-kserve-997cc74d9-pn62s"] Sep 16 14:07:40.537037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:40.536994 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:41.471595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:41.471560 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09aae84e-476b-410d-98f9-86f02fc5823b" path="/var/lib/kubelet/pods/09aae84e-476b-410d-98f9-86f02fc5823b/volumes" Sep 16 14:07:41.472154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:41.472134 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326b971e-f46a-4b38-8eca-c25fda559120" path="/var/lib/kubelet/pods/326b971e-f46a-4b38-8eca-c25fda559120/volumes" Sep 16 14:07:41.472536 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:41.472518 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" path="/var/lib/kubelet/pods/3a82cfad-f3c8-4241-bfb4-137d4d8e5218/volumes" Sep 16 14:07:41.472983 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:41.472957 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" path="/var/lib/kubelet/pods/a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff/volumes" Sep 16 14:07:41.899485 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:41.899394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:42.955764 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:42.955727 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:44.154003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:44.153966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:44.303142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:44.303100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:44.923958 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:44.923922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:45.101114 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.101078 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:45.135266 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135236 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:07:45.135674 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135660 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" containerName="main" Sep 16 14:07:45.135725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135676 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" containerName="main" Sep 16 14:07:45.135725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135691 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09aae84e-476b-410d-98f9-86f02fc5823b" containerName="main" Sep 16 14:07:45.135725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135698 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09aae84e-476b-410d-98f9-86f02fc5823b" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135726 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326b971e-f46a-4b38-8eca-c25fda559120" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135732 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="326b971e-f46a-4b38-8eca-c25fda559120" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135742 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135747 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135813 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="326b971e-f46a-4b38-8eca-c25fda559120" containerName="main" Sep 16 14:07:45.135821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135820 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a241c8ef-0ced-41ce-acfd-e5c34c7dc2ff" containerName="main" Sep 16 14:07:45.136026 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135826 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09aae84e-476b-410d-98f9-86f02fc5823b" containerName="main" Sep 16 14:07:45.136026 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.135834 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a82cfad-f3c8-4241-bfb4-137d4d8e5218" containerName="main" Sep 16 14:07:45.140589 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.140574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.146523 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.146502 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8a4447-dockercfg-lb9mb\"" Sep 16 14:07:45.146632 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.146505 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8a4447\"" Sep 16 14:07:45.181823 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.181753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:07:45.249611 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.249579 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:07:45.253093 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.253078 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.285481 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.285454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.285638 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.285486 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6q6\" (UniqueName: \"kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.289263 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.289230 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:07:45.386451 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.386415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.386451 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.386449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6q6\" (UniqueName: \"kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.386658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.386502 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.386658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.386542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqfx\" (UniqueName: \"kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.387147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.387129 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.402769 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.402741 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6q6\" (UniqueName: \"kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6\") pod \"payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.450401 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.450370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:45.488189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.488149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.488327 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.488236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqfx\" (UniqueName: \"kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.488825 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.488804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.509745 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.509716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqfx\" (UniqueName: \"kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx\") pod \"payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.563254 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.563214 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:45.581015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.580987 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8"] Sep 16 14:07:45.594652 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.594614 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-19203d-29826120-z79d8"] Sep 16 14:07:45.643386 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.643354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:45.760663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.760569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:07:45.764674 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:07:45.764644 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea68f94f_a2b3_474b_b50e_1c4b98314a84.slice/crio-74da986457fe8e2a3388601972a2728547bc33f44121891b6429f0e2421da9b9 WatchSource:0}: Error finding container 74da986457fe8e2a3388601972a2728547bc33f44121891b6429f0e2421da9b9: Status 404 returned error can't find the container with id 74da986457fe8e2a3388601972a2728547bc33f44121891b6429f0e2421da9b9 Sep 16 14:07:45.771111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.771089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:45.853747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.853717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:07:45.855737 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:07:45.855701 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eaf26af_871d_4e57_814d_d358bdc50281.slice/crio-1f078a72e0e217c85ef968a23a3f982a07dadedc9c78d3cca117dcc63f15f45b WatchSource:0}: Error finding container 1f078a72e0e217c85ef968a23a3f982a07dadedc9c78d3cca117dcc63f15f45b: Status 404 returned error can't find the container with id 1f078a72e0e217c85ef968a23a3f982a07dadedc9c78d3cca117dcc63f15f45b Sep 16 14:07:45.908896 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:45.908856 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:46.010126 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.010085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:46.196842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.196807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:46.214937 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.214897 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" event={"ID":"ea68f94f-a2b3-474b-b50e-1c4b98314a84","Type":"ContainerStarted","Data":"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a"} Sep 16 14:07:46.215130 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.214963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" event={"ID":"ea68f94f-a2b3-474b-b50e-1c4b98314a84","Type":"ContainerStarted","Data":"74da986457fe8e2a3388601972a2728547bc33f44121891b6429f0e2421da9b9"} Sep 16 14:07:46.215130 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.215095 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:46.216512 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.216491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" event={"ID":"8eaf26af-871d-4e57-814d-d358bdc50281","Type":"ContainerStarted","Data":"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511"} Sep 16 14:07:46.216598 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.216533 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" event={"ID":"8eaf26af-871d-4e57-814d-d358bdc50281","Type":"ContainerStarted","Data":"1f078a72e0e217c85ef968a23a3f982a07dadedc9c78d3cca117dcc63f15f45b"} Sep 16 14:07:46.216598 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.216555 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:46.294861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.294801 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" podStartSLOduration=1.294778952 podStartE2EDuration="1.294778952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:46.259791335 +0000 UTC m=+1485.300385289" watchObservedRunningTime="2026-09-16 14:07:46.294778952 +0000 UTC m=+1485.335372908" Sep 16 14:07:46.295196 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.295169 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" podStartSLOduration=1.295158149 podStartE2EDuration="1.295158149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:46.293400115 +0000 UTC m=+1485.333994069" watchObservedRunningTime="2026-09-16 14:07:46.295158149 +0000 UTC m=+1485.335752103" Sep 16 14:07:46.582050 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:46.581954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:47.429591 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:47.429554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:47.472184 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:47.472152 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79613e6c-8a7f-42f1-b3ac-19eb77c8189c" path="/var/lib/kubelet/pods/79613e6c-8a7f-42f1-b3ac-19eb77c8189c/volumes" Sep 16 14:07:48.843857 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:48.843821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:50.054754 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:50.054722 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:51.973084 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:51.973043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:52.174889 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:52.174836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:53.056265 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:53.056216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:53.693030 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:53.692998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:53.791347 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:53.791316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:54.194304 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:54.194274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:54.298326 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:54.298291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:54.361214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:54.361179 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:07:56.102644 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:56.102610 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:57.224084 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:57.224045 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:07:57.224482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:57.224117 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:07:57.947895 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:57.947839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:58.190169 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:58.190135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:59.115436 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:59.115402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:59.528354 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:07:59.528319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:00.085531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:00.085494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:01.115608 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:01.115568 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:01.767437 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:01.767402 2559 scope.go:117] "RemoveContainer" containerID="188a47f9d017684991f2e151fe6239fbd48ce6563db6d176371ade219528fd4e" Sep 16 14:08:01.778265 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:01.778239 2559 scope.go:117] "RemoveContainer" containerID="25c78db7dec2eb8f6b0e2041bfac993c422ec3d298af332b3fa027c70d12f317" Sep 16 14:08:01.931324 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:01.931287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:02.112134 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:02.112039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:02.219123 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:02.219090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:03.045940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:03.045904 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:03.625265 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:03.625228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:04.017346 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:04.017311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:05.040332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:05.040289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:05.927971 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:05.927934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:06.725177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:06.725143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:07.672305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:07.672268 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:10.452522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:10.452482 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:12.508929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:12.508888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:14.300780 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:14.300742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:15.744353 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:15.744314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:08:18.262154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.262120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:18.317343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.317310 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:08:18.317631 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.317594 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" podUID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" containerName="payload-processing" containerID="cri-o://d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0" gracePeriod=30 Sep 16 14:08:18.326538 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.326511 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:08:18.326767 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.326741 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" podUID="adf641f6-b35c-4f10-8513-e0668791b63b" containerName="payload-pre-processing" containerID="cri-o://bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4" gracePeriod=30 Sep 16 14:08:18.594558 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.594534 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:08:18.597697 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.597675 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:08:18.695466 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695429 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bctg\" (UniqueName: \"kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg\") pod \"adf641f6-b35c-4f10-8513-e0668791b63b\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " Sep 16 14:08:18.695640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume\") pod \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " Sep 16 14:08:18.695640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695518 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume\") pod \"adf641f6-b35c-4f10-8513-e0668791b63b\" (UID: \"adf641f6-b35c-4f10-8513-e0668791b63b\") " Sep 16 14:08:18.695640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695558 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfdx4\" (UniqueName: \"kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4\") pod \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\" (UID: \"6b5d3caa-cdfe-455c-978b-e80ca972daa6\") " Sep 16 14:08:18.696013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adf641f6-b35c-4f10-8513-e0668791b63b" (UID: "adf641f6-b35c-4f10-8513-e0668791b63b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:18.696013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.695977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b5d3caa-cdfe-455c-978b-e80ca972daa6" (UID: "6b5d3caa-cdfe-455c-978b-e80ca972daa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:18.697577 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.697559 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg" (OuterVolumeSpecName: "kube-api-access-8bctg") pod "adf641f6-b35c-4f10-8513-e0668791b63b" (UID: "adf641f6-b35c-4f10-8513-e0668791b63b"). InnerVolumeSpecName "kube-api-access-8bctg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:18.697709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.697687 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4" (OuterVolumeSpecName: "kube-api-access-gfdx4") pod "6b5d3caa-cdfe-455c-978b-e80ca972daa6" (UID: "6b5d3caa-cdfe-455c-978b-e80ca972daa6"). InnerVolumeSpecName "kube-api-access-gfdx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:18.797013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.796924 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5d3caa-cdfe-455c-978b-e80ca972daa6-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.797013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.796955 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf641f6-b35c-4f10-8513-e0668791b63b-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.797013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.796966 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfdx4\" (UniqueName: \"kubernetes.io/projected/6b5d3caa-cdfe-455c-978b-e80ca972daa6-kube-api-access-gfdx4\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.797013 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:18.796975 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bctg\" (UniqueName: \"kubernetes.io/projected/adf641f6-b35c-4f10-8513-e0668791b63b-kube-api-access-8bctg\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:19.391923 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.391887 2559 generic.go:356] "Generic (PLEG): container finished" podID="adf641f6-b35c-4f10-8513-e0668791b63b" containerID="bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4" exitCode=0 Sep 16 14:08:19.392373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.391948 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" Sep 16 14:08:19.392373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.391955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" event={"ID":"adf641f6-b35c-4f10-8513-e0668791b63b","Type":"ContainerDied","Data":"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4"} Sep 16 14:08:19.392373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.392040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp" event={"ID":"adf641f6-b35c-4f10-8513-e0668791b63b","Type":"ContainerDied","Data":"8cc29d25a18677affdd7725e69941cf19b104a080bfdeefdaa4a2688674198a7"} Sep 16 14:08:19.392373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.392069 2559 scope.go:117] "RemoveContainer" containerID="bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4" Sep 16 14:08:19.393614 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.393594 2559 generic.go:356] "Generic (PLEG): container finished" podID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" containerID="d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0" exitCode=0 Sep 16 14:08:19.393725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.393644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" event={"ID":"6b5d3caa-cdfe-455c-978b-e80ca972daa6","Type":"ContainerDied","Data":"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0"} Sep 16 14:08:19.393725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.393660 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" Sep 16 14:08:19.393725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.393700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt" event={"ID":"6b5d3caa-cdfe-455c-978b-e80ca972daa6","Type":"ContainerDied","Data":"12ac43b3a6c75367c006fd5157998a7a111ebc476fc3b95d69d42f86bc9ccaaa"} Sep 16 14:08:19.405995 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.405541 2559 scope.go:117] "RemoveContainer" containerID="bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4" Sep 16 14:08:19.406490 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:19.406465 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4\": container with ID starting with bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4 not found: ID does not exist" containerID="bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4" Sep 16 14:08:19.406605 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.406498 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4"} err="failed to get container status \"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4\": rpc error: code = NotFound desc = could not find container \"bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4\": container with ID starting with bffddfda5bc3f35b2a459d1429d307004d32b4aa6631d3fda150f699f28256f4 not found: ID does not exist" Sep 16 14:08:19.406605 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.406519 2559 scope.go:117] "RemoveContainer" containerID="d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0" Sep 16 14:08:19.414387 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.414370 2559 scope.go:117] "RemoveContainer" containerID="d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0" Sep 16 14:08:19.414630 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:19.414612 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0\": container with ID starting with d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0 not found: ID does not exist" containerID="d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0" Sep 16 14:08:19.414677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.414636 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0"} err="failed to get container status \"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0\": rpc error: code = NotFound desc = could not find container \"d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0\": container with ID starting with d73483550ba4dd20aebedd4c02971d1e27b8ada1b126388e19d9dff6be5b2ec0 not found: ID does not exist" Sep 16 14:08:19.425297 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.425275 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:08:19.426987 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.426969 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f75a3-69b568848b-9pbvp"] Sep 16 14:08:19.444279 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.444258 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:08:19.453020 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.453000 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f75a3-c4b9d476f-hmfnt"] Sep 16 14:08:19.471491 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.471473 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" path="/var/lib/kubelet/pods/6b5d3caa-cdfe-455c-978b-e80ca972daa6/volumes" Sep 16 14:08:19.471828 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:19.471815 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf641f6-b35c-4f10-8513-e0668791b63b" path="/var/lib/kubelet/pods/adf641f6-b35c-4f10-8513-e0668791b63b/volumes" Sep 16 14:08:21.036302 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:21.036271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:21.288138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:21.288055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:22.074308 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:22.074260 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:22.997839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:22.997804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:25.524525 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:25.524450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:25.644488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:25.644440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:25.742998 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:25.742955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:26.120045 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120009 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl"] Sep 16 14:08:26.120378 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120365 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" containerName="payload-processing" Sep 16 14:08:26.120419 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120379 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" containerName="payload-processing" Sep 16 14:08:26.120419 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120389 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf641f6-b35c-4f10-8513-e0668791b63b" containerName="payload-pre-processing" Sep 16 14:08:26.120419 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120395 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf641f6-b35c-4f10-8513-e0668791b63b" containerName="payload-pre-processing" Sep 16 14:08:26.120515 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120473 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf641f6-b35c-4f10-8513-e0668791b63b" containerName="payload-pre-processing" Sep 16 14:08:26.120515 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.120483 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5d3caa-cdfe-455c-978b-e80ca972daa6" containerName="payload-processing" Sep 16 14:08:26.130171 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.129763 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:26.132305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.132270 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:08:26.132472 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.132452 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:08:26.132785 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.132274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl"] Sep 16 14:08:26.262611 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.262566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl\" (UID: \"ce66dee2-609a-48eb-9a2a-ef63c6537e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:26.364157 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.364114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl\" (UID: \"ce66dee2-609a-48eb-9a2a-ef63c6537e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:26.364752 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.364728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl\" (UID: \"ce66dee2-609a-48eb-9a2a-ef63c6537e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:26.442832 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.442740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:26.717472 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.717446 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl"] Sep 16 14:08:26.720078 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:08:26.720051 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce66dee2_609a_48eb_9a2a_ef63c6537e1a.slice/crio-0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10 WatchSource:0}: Error finding container 0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10: Status 404 returned error can't find the container with id 0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10 Sep 16 14:08:26.994185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:26.994147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:27.437141 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:27.437048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" event={"ID":"ce66dee2-609a-48eb-9a2a-ef63c6537e1a","Type":"ContainerStarted","Data":"eb7509030383561a418cbd4c184299829bfdf42009e5db2527128701475da255"} Sep 16 14:08:27.437141 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:27.437117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" event={"ID":"ce66dee2-609a-48eb-9a2a-ef63c6537e1a","Type":"ContainerStarted","Data":"0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10"} Sep 16 14:08:27.454603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:27.454558 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" podStartSLOduration=1.454543465 podStartE2EDuration="1.454543465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:27.452819919 +0000 UTC m=+1526.493413886" watchObservedRunningTime="2026-09-16 14:08:27.454543465 +0000 UTC m=+1526.495137419" Sep 16 14:08:27.699859 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:27.699828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:27.800719 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:27.800687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:28.445505 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.445472 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce66dee2-609a-48eb-9a2a-ef63c6537e1a" containerID="eb7509030383561a418cbd4c184299829bfdf42009e5db2527128701475da255" exitCode=0 Sep 16 14:08:28.445687 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.445537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" event={"ID":"ce66dee2-609a-48eb-9a2a-ef63c6537e1a","Type":"ContainerDied","Data":"eb7509030383561a418cbd4c184299829bfdf42009e5db2527128701475da255"} Sep 16 14:08:28.899805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.899718 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-f8f684"] Sep 16 14:08:28.903201 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.903184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:28.905795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.905768 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:08:28.908170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.908146 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-f8f684"] Sep 16 14:08:28.988931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:28.988903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rh4\" (UniqueName: \"kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4\") pod \"test-curl-7899-f8f684\" (UID: \"cce8200f-ca2d-485b-9fe8-49890cd1bf0f\") " pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:29.048245 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.048210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:29.089985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.089947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rh4\" (UniqueName: \"kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4\") pod \"test-curl-7899-f8f684\" (UID: \"cce8200f-ca2d-485b-9fe8-49890cd1bf0f\") " pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:29.106241 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.106216 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rh4\" (UniqueName: \"kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4\") pod \"test-curl-7899-f8f684\" (UID: \"cce8200f-ca2d-485b-9fe8-49890cd1bf0f\") " pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:29.214012 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.213974 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:29.485067 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.485006 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-f8f684"] Sep 16 14:08:29.488230 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:08:29.488197 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcce8200f_ca2d_485b_9fe8_49890cd1bf0f.slice/crio-e0b3ab6a16fd6b7de6d754225ebc5e5d4453cc58c78c7e8d9fde20b96f7603d7 WatchSource:0}: Error finding container e0b3ab6a16fd6b7de6d754225ebc5e5d4453cc58c78c7e8d9fde20b96f7603d7: Status 404 returned error can't find the container with id e0b3ab6a16fd6b7de6d754225ebc5e5d4453cc58c78c7e8d9fde20b96f7603d7 Sep 16 14:08:29.543833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.543804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:29.576746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.576723 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:29.695245 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.695207 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca\") pod \"ce66dee2-609a-48eb-9a2a-ef63c6537e1a\" (UID: \"ce66dee2-609a-48eb-9a2a-ef63c6537e1a\") " Sep 16 14:08:29.695582 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.695563 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce66dee2-609a-48eb-9a2a-ef63c6537e1a" (UID: "ce66dee2-609a-48eb-9a2a-ef63c6537e1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:29.796864 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:29.796776 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce66dee2-609a-48eb-9a2a-ef63c6537e1a-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:30.458931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.458885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-f8f684" event={"ID":"cce8200f-ca2d-485b-9fe8-49890cd1bf0f","Type":"ContainerStarted","Data":"9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2"} Sep 16 14:08:30.458931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.458935 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-f8f684" event={"ID":"cce8200f-ca2d-485b-9fe8-49890cd1bf0f","Type":"ContainerStarted","Data":"e0b3ab6a16fd6b7de6d754225ebc5e5d4453cc58c78c7e8d9fde20b96f7603d7"} Sep 16 14:08:30.460419 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.460396 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" Sep 16 14:08:30.460525 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.460446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl" event={"ID":"ce66dee2-609a-48eb-9a2a-ef63c6537e1a","Type":"ContainerDied","Data":"0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10"} Sep 16 14:08:30.460525 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.460514 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba841b324821a68a14a4aada8dd0a7b9bac89801fa6365e4e0504a3c826bb10" Sep 16 14:08:30.476437 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.476386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7899-f8f684" podStartSLOduration=2.476371378 podStartE2EDuration="2.476371378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:30.474331823 +0000 UTC m=+1529.514925777" watchObservedRunningTime="2026-09-16 14:08:30.476371378 +0000 UTC m=+1529.516965331" Sep 16 14:08:30.477219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:30.477196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:31.064795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:31.064755 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-f8f684"] Sep 16 14:08:31.066962 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:31.066928 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-f8f684"] Sep 16 14:08:31.467587 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:31.467555 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7899-f8f684" container="test-curl-7899-f8f684" Sep 16 14:08:31.473255 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:31.473227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:32.469888 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:32.469823 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7899-f8f684" podUID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" containerName="test-curl-7899-f8f684" containerID="cri-o://9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2" gracePeriod=2 Sep 16 14:08:32.556486 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:32.556420 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:32.746285 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:32.746202 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:08:33.426277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:33.426244 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:08:33.448412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:33.448383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:34.653007 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:34.652985 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:34.747195 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:34.747116 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7rh4\" (UniqueName: \"kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4\") pod \"cce8200f-ca2d-485b-9fe8-49890cd1bf0f\" (UID: \"cce8200f-ca2d-485b-9fe8-49890cd1bf0f\") " Sep 16 14:08:34.749150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:34.749123 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4" (OuterVolumeSpecName: "kube-api-access-t7rh4") pod "cce8200f-ca2d-485b-9fe8-49890cd1bf0f" (UID: "cce8200f-ca2d-485b-9fe8-49890cd1bf0f"). InnerVolumeSpecName "kube-api-access-t7rh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:34.848559 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:34.848525 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rh4\" (UniqueName: \"kubernetes.io/projected/cce8200f-ca2d-485b-9fe8-49890cd1bf0f-kube-api-access-t7rh4\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:35.471662 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.471628 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" path="/var/lib/kubelet/pods/cce8200f-ca2d-485b-9fe8-49890cd1bf0f/volumes" Sep 16 14:08:35.485561 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.485538 2559 generic.go:356] "Generic (PLEG): container finished" podID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" containerID="9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2" exitCode=137 Sep 16 14:08:35.485667 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.485574 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-f8f684" Sep 16 14:08:35.485667 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.485605 2559 scope.go:117] "RemoveContainer" containerID="9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2" Sep 16 14:08:35.494450 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.494432 2559 scope.go:117] "RemoveContainer" containerID="9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2" Sep 16 14:08:35.494711 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:35.494691 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2\": container with ID starting with 9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2 not found: ID does not exist" containerID="9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2" Sep 16 14:08:35.494779 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:35.494719 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2"} err="failed to get container status \"9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2\": rpc error: code = NotFound desc = could not find container \"9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2\": container with ID starting with 9bbf730cfdc101b660c8bfcd77eba4a11c89409af546b0436511726902295cf2 not found: ID does not exist" Sep 16 14:08:40.047435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047402 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:08:40.047820 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047806 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce66dee2-609a-48eb-9a2a-ef63c6537e1a" containerName="revoke-keys" Sep 16 14:08:40.047879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047821 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce66dee2-609a-48eb-9a2a-ef63c6537e1a" containerName="revoke-keys" Sep 16 14:08:40.047879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047832 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" containerName="test-curl-7899-f8f684" Sep 16 14:08:40.047879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047838 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" containerName="test-curl-7899-f8f684" Sep 16 14:08:40.047978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047917 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce66dee2-609a-48eb-9a2a-ef63c6537e1a" containerName="revoke-keys" Sep 16 14:08:40.047978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.047929 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce8200f-ca2d-485b-9fe8-49890cd1bf0f" containerName="test-curl-7899-f8f684" Sep 16 14:08:40.051989 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.051967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.054352 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.054329 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"openshift-service-ca.crt\"" Sep 16 14:08:40.055248 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.055227 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"default-dockercfg-7g6qm\"" Sep 16 14:08:40.055248 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.055240 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"kube-root-ca.crt\"" Sep 16 14:08:40.055377 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.055247 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"sub-test-model-46224d-kserve-self-signed-certs\"" Sep 16 14:08:40.060929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.060902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:08:40.076559 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.076530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:40.096797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.096938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.096938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.096938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdpb\" (UniqueName: \"kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.096938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.096938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.096939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197501 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.197711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdpb\" (UniqueName: \"kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.198079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.198079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.198079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.197991 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.199863 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.199831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.200102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.200083 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.207050 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.207026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdpb\" (UniqueName: \"kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb\") pod \"sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.361532 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.361444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:40.646153 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:40.646128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:08:40.648299 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:08:40.648274 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c91edd_6bce_4a8e_ba09_2fb57f0ae4f6.slice/crio-26e9b30c7ad76a737cc822948bf2b24eba0188dd0343314c5a3897456b84af9d WatchSource:0}: Error finding container 26e9b30c7ad76a737cc822948bf2b24eba0188dd0343314c5a3897456b84af9d: Status 404 returned error can't find the container with id 26e9b30c7ad76a737cc822948bf2b24eba0188dd0343314c5a3897456b84af9d Sep 16 14:08:41.521033 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.520991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" event={"ID":"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6","Type":"ContainerStarted","Data":"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e"} Sep 16 14:08:41.521415 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.521043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" event={"ID":"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6","Type":"ContainerStarted","Data":"26e9b30c7ad76a737cc822948bf2b24eba0188dd0343314c5a3897456b84af9d"} Sep 16 14:08:41.542495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.542445 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" podStartSLOduration=1.542429791 podStartE2EDuration="1.542429791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:41.541120901 +0000 UTC m=+1540.581714856" watchObservedRunningTime="2026-09-16 14:08:41.542429791 +0000 UTC m=+1540.583023745" Sep 16 14:08:41.629085 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.629045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:41.658600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.658562 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:08:41.658906 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.658857 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" podUID="8eaf26af-871d-4e57-814d-d358bdc50281" containerName="payload-processing" containerID="cri-o://c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511" gracePeriod=30 Sep 16 14:08:41.668214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.667960 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:08:41.668214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.668184 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" podUID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" containerName="payload-pre-processing" containerID="cri-o://8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a" gracePeriod=30 Sep 16 14:08:41.700548 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.700515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:41.798860 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.798818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:41.902653 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.902616 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl"] Sep 16 14:08:41.904958 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.904931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:41.907057 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.907008 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a4447-kr5sl"] Sep 16 14:08:41.940185 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.940164 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:08:41.945082 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.945062 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:08:41.996691 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:41.996657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:42.012932 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.012898 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn6q6\" (UniqueName: \"kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6\") pod \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " Sep 16 14:08:42.013118 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.012940 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume\") pod \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\" (UID: \"ea68f94f-a2b3-474b-b50e-1c4b98314a84\") " Sep 16 14:08:42.013118 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.013038 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjqfx\" (UniqueName: \"kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx\") pod \"8eaf26af-871d-4e57-814d-d358bdc50281\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " Sep 16 14:08:42.013416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.013386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea68f94f-a2b3-474b-b50e-1c4b98314a84" (UID: "ea68f94f-a2b3-474b-b50e-1c4b98314a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:42.015718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.015632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6" (OuterVolumeSpecName: "kube-api-access-dn6q6") pod "ea68f94f-a2b3-474b-b50e-1c4b98314a84" (UID: "ea68f94f-a2b3-474b-b50e-1c4b98314a84"). InnerVolumeSpecName "kube-api-access-dn6q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:42.016018 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.015988 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx" (OuterVolumeSpecName: "kube-api-access-hjqfx") pod "8eaf26af-871d-4e57-814d-d358bdc50281" (UID: "8eaf26af-871d-4e57-814d-d358bdc50281"). InnerVolumeSpecName "kube-api-access-hjqfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:42.114277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.114187 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume\") pod \"8eaf26af-871d-4e57-814d-d358bdc50281\" (UID: \"8eaf26af-871d-4e57-814d-d358bdc50281\") " Sep 16 14:08:42.114431 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.114410 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6q6\" (UniqueName: \"kubernetes.io/projected/ea68f94f-a2b3-474b-b50e-1c4b98314a84-kube-api-access-dn6q6\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.114431 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.114424 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea68f94f-a2b3-474b-b50e-1c4b98314a84-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.114511 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.114433 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqfx\" (UniqueName: \"kubernetes.io/projected/8eaf26af-871d-4e57-814d-d358bdc50281-kube-api-access-hjqfx\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.114560 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.114536 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8eaf26af-871d-4e57-814d-d358bdc50281" (UID: "8eaf26af-871d-4e57-814d-d358bdc50281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:42.214924 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.214890 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eaf26af-871d-4e57-814d-d358bdc50281-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.353422 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.353383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:42.458977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.458942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:42.527617 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.527579 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" containerID="8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a" exitCode=0 Sep 16 14:08:42.528037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.527621 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" event={"ID":"ea68f94f-a2b3-474b-b50e-1c4b98314a84","Type":"ContainerDied","Data":"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a"} Sep 16 14:08:42.528037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.527679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" event={"ID":"ea68f94f-a2b3-474b-b50e-1c4b98314a84","Type":"ContainerDied","Data":"74da986457fe8e2a3388601972a2728547bc33f44121891b6429f0e2421da9b9"} Sep 16 14:08:42.528037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.527703 2559 scope.go:117] "RemoveContainer" containerID="8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a" Sep 16 14:08:42.528037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.527628 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5" Sep 16 14:08:42.529160 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.529138 2559 generic.go:356] "Generic (PLEG): container finished" podID="8eaf26af-871d-4e57-814d-d358bdc50281" containerID="c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511" exitCode=0 Sep 16 14:08:42.529270 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.529182 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" Sep 16 14:08:42.529270 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.529224 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" event={"ID":"8eaf26af-871d-4e57-814d-d358bdc50281","Type":"ContainerDied","Data":"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511"} Sep 16 14:08:42.529364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.529275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh" event={"ID":"8eaf26af-871d-4e57-814d-d358bdc50281","Type":"ContainerDied","Data":"1f078a72e0e217c85ef968a23a3f982a07dadedc9c78d3cca117dcc63f15f45b"} Sep 16 14:08:42.538706 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.538685 2559 scope.go:117] "RemoveContainer" containerID="8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a" Sep 16 14:08:42.539099 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:42.539026 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a\": container with ID starting with 8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a not found: ID does not exist" containerID="8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a" Sep 16 14:08:42.539099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.539052 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a"} err="failed to get container status \"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a\": rpc error: code = NotFound desc = could not find container \"8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a\": container with ID starting with 8653c027b422961cc701e0a58b41b1e8a3e641dda4937d34bff69c7fcc0d087a not found: ID does not exist" Sep 16 14:08:42.539099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.539069 2559 scope.go:117] "RemoveContainer" containerID="c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511" Sep 16 14:08:42.547215 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.547193 2559 scope.go:117] "RemoveContainer" containerID="c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511" Sep 16 14:08:42.547526 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:08:42.547504 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511\": container with ID starting with c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511 not found: ID does not exist" containerID="c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511" Sep 16 14:08:42.547586 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.547536 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511"} err="failed to get container status \"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511\": rpc error: code = NotFound desc = could not find container \"c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511\": container with ID starting with c6e0e3293c9ac6d6ea71e087dc284be4fd8e8736de8e56cd19e7da0d9d437511 not found: ID does not exist" Sep 16 14:08:42.552364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.552335 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:08:42.557424 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.557396 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8a4447-76964bb44b-l7tz5"] Sep 16 14:08:42.569432 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.569405 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:08:42.574407 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:42.574374 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8a4447-6957849dfc-ghmsh"] Sep 16 14:08:43.283812 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:43.283778 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:43.472163 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:43.472134 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eaf26af-871d-4e57-814d-d358bdc50281" path="/var/lib/kubelet/pods/8eaf26af-871d-4e57-814d-d358bdc50281/volumes" Sep 16 14:08:43.472527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:43.472513 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce66dee2-609a-48eb-9a2a-ef63c6537e1a" path="/var/lib/kubelet/pods/ce66dee2-609a-48eb-9a2a-ef63c6537e1a/volumes" Sep 16 14:08:43.472858 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:43.472845 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" path="/var/lib/kubelet/pods/ea68f94f-a2b3-474b-b50e-1c4b98314a84/volumes" Sep 16 14:08:44.109130 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:44.109099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:44.934841 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:44.934782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:45.441699 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:45.441660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:08:46.033723 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:46.033679 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.210592 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.210559 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.444580 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.444534 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" podUID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" containerName="main" containerID="cri-o://c4c1300348ba1f009bb471cb1e726b5d1d4cea39416c9d79c5ba602c4aa440a3" gracePeriod=45 Sep 16 14:08:48.449279 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.449246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.563335 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.563307 2559 generic.go:356] "Generic (PLEG): container finished" podID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" containerID="c4c1300348ba1f009bb471cb1e726b5d1d4cea39416c9d79c5ba602c4aa440a3" exitCode=0 Sep 16 14:08:48.563462 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.563378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" event={"ID":"2a96e849-dc7b-4c91-939d-8f2bdca87a73","Type":"ContainerDied","Data":"c4c1300348ba1f009bb471cb1e726b5d1d4cea39416c9d79c5ba602c4aa440a3"} Sep 16 14:08:48.695668 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.695642 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:08:48.775568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775531 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6ttc\" (UniqueName: \"kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775744 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775585 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775744 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775744 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775681 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775744 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775950 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir\") pod \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\" (UID: \"2a96e849-dc7b-4c91-939d-8f2bdca87a73\") " Sep 16 14:08:48.775996 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.775943 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:08:48.776098 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.776070 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:08:48.776218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.776099 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.776218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.776175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home" (OuterVolumeSpecName: "home") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:08:48.777901 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.777860 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:08:48.777988 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.777894 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc" (OuterVolumeSpecName: "kube-api-access-r6ttc") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "kube-api-access-r6ttc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:48.777988 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.777902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm" (OuterVolumeSpecName: "dshm") pod "2a96e849-dc7b-4c91-939d-8f2bdca87a73" (UID: "2a96e849-dc7b-4c91-939d-8f2bdca87a73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:08:48.820880 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.820769 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:48.876909 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.876848 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ttc\" (UniqueName: \"kubernetes.io/projected/2a96e849-dc7b-4c91-939d-8f2bdca87a73-kube-api-access-r6ttc\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.876909 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.876905 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.876909 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.876919 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.877157 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.876930 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.877157 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.876940 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96e849-dc7b-4c91-939d-8f2bdca87a73-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:48.925600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:48.925571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:49.079472 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.079397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:49.457107 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457069 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw"] Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457467 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" containerName="main" Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457479 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" containerName="main" Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457491 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" containerName="payload-pre-processing" Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457497 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" containerName="payload-pre-processing" Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457507 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eaf26af-871d-4e57-814d-d358bdc50281" containerName="payload-processing" Sep 16 14:08:49.457521 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457514 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eaf26af-871d-4e57-814d-d358bdc50281" containerName="payload-processing" Sep 16 14:08:49.457718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457573 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eaf26af-871d-4e57-814d-d358bdc50281" containerName="payload-processing" Sep 16 14:08:49.457718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457583 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea68f94f-a2b3-474b-b50e-1c4b98314a84" containerName="payload-pre-processing" Sep 16 14:08:49.457718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.457590 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" containerName="main" Sep 16 14:08:49.460762 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.460741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:49.463992 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.463969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:08:49.468036 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.467857 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:08:49.480729 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.480705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw\" (UID: \"937cc127-d932-4d23-a793-971838700215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:49.483926 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.483900 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw"] Sep 16 14:08:49.570293 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.570256 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" Sep 16 14:08:49.570482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.570296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c" event={"ID":"2a96e849-dc7b-4c91-939d-8f2bdca87a73","Type":"ContainerDied","Data":"a8345b0a4fa3b52977dd994b9faf6924c10bdf66a68803b62bb48fe3fa0ac148"} Sep 16 14:08:49.570482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.570345 2559 scope.go:117] "RemoveContainer" containerID="c4c1300348ba1f009bb471cb1e726b5d1d4cea39416c9d79c5ba602c4aa440a3" Sep 16 14:08:49.581679 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.581652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw\" (UID: \"937cc127-d932-4d23-a793-971838700215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:49.582198 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.582181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw\" (UID: \"937cc127-d932-4d23-a793-971838700215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:49.610631 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.610602 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:08:49.614208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.614180 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/auth-test-model-b4884e-kserve-7b8fc7566c-l522c"] Sep 16 14:08:49.736374 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.736297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:49.771964 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:49.771935 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:50.051791 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.051764 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw"] Sep 16 14:08:50.053567 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:08:50.053539 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937cc127_d932_4d23_a793_971838700215.slice/crio-62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90 WatchSource:0}: Error finding container 62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90: Status 404 returned error can't find the container with id 62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90 Sep 16 14:08:50.362565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.362481 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:50.362565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.362520 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:50.374551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.374525 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:50.577899 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.577827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" event={"ID":"937cc127-d932-4d23-a793-971838700215","Type":"ContainerStarted","Data":"d03496c0469c012c9481efd8081ee211013a0ab7017d985a9c210ef411d41728"} Sep 16 14:08:50.578456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.577916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" event={"ID":"937cc127-d932-4d23-a793-971838700215","Type":"ContainerStarted","Data":"62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90"} Sep 16 14:08:50.589382 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.589353 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:08:50.598590 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.598545 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" podStartSLOduration=1.5985326899999999 podStartE2EDuration="1.59853269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:50.596927049 +0000 UTC m=+1549.637521002" watchObservedRunningTime="2026-09-16 14:08:50.59853269 +0000 UTC m=+1549.639126676" Sep 16 14:08:50.670959 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:50.670827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:51.471688 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:51.471643 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a96e849-dc7b-4c91-939d-8f2bdca87a73" path="/var/lib/kubelet/pods/2a96e849-dc7b-4c91-939d-8f2bdca87a73/volumes" Sep 16 14:08:51.584173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:51.584131 2559 generic.go:356] "Generic (PLEG): container finished" podID="937cc127-d932-4d23-a793-971838700215" containerID="d03496c0469c012c9481efd8081ee211013a0ab7017d985a9c210ef411d41728" exitCode=0 Sep 16 14:08:51.584529 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:51.584217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" event={"ID":"937cc127-d932-4d23-a793-971838700215","Type":"ContainerDied","Data":"d03496c0469c012c9481efd8081ee211013a0ab7017d985a9c210ef411d41728"} Sep 16 14:08:51.807497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:51.807403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.715464 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:52.715441 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:52.727060 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:52.727024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:52.812453 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:52.812401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca\") pod \"937cc127-d932-4d23-a793-971838700215\" (UID: \"937cc127-d932-4d23-a793-971838700215\") " Sep 16 14:08:52.812813 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:52.812789 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "937cc127-d932-4d23-a793-971838700215" (UID: "937cc127-d932-4d23-a793-971838700215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:52.913239 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:52.913134 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/937cc127-d932-4d23-a793-971838700215-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:08:53.555059 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:53.555012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:53.595165 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:53.595132 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" Sep 16 14:08:53.595348 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:53.595166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw" event={"ID":"937cc127-d932-4d23-a793-971838700215","Type":"ContainerDied","Data":"62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90"} Sep 16 14:08:53.595348 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:53.595319 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a7bfc73d06f3b37741501f8c0a491396989c7d404a779f7e4e3b256e328d90" Sep 16 14:08:53.639582 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:53.639545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:54.599076 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:54.599047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:54.917809 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:54.917713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:59.039149 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.039110 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:08:59.039704 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.039502 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937cc127-d932-4d23-a793-971838700215" containerName="revoke-keys" Sep 16 14:08:59.039704 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.039513 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="937cc127-d932-4d23-a793-971838700215" containerName="revoke-keys" Sep 16 14:08:59.039704 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.039589 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="937cc127-d932-4d23-a793-971838700215" containerName="revoke-keys" Sep 16 14:08:59.042344 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.042325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.044876 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.044842 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"sub-test-model-b4884e-kserve-self-signed-certs\"" Sep 16 14:08:59.045011 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.044990 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"openshift-service-ca.crt\"" Sep 16 14:08:59.045660 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.045639 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"kube-root-ca.crt\"" Sep 16 14:08:59.045753 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.045663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"default-dockercfg-7k44r\"" Sep 16 14:08:59.052219 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.052198 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:08:59.070475 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.070447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:59.168839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.168798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4v6\" (UniqueName: \"kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.168839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.168841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.169099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.168893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.169099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.168947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.169099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.168973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.169099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.169019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270403 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270403 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270409 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270673 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4v6\" (UniqueName: \"kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270903 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.270903 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.271011 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.270945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.272689 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.272665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.272849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.272832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.280096 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.280075 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4v6\" (UniqueName: \"kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6\") pod \"sub-test-model-b4884e-kserve-69c95c69fc-9gnbb\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.352483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.352379 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:08:59.625305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:08:59.625131 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:08:59.628312 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:08:59.628285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27ec3ce4_0b08_4f31_be7c_105b26e9a830.slice/crio-422990c1f5789313630f2394c20a7c8f5f9b2a72353c225f6961dac6df8375a8 WatchSource:0}: Error finding container 422990c1f5789313630f2394c20a7c8f5f9b2a72353c225f6961dac6df8375a8: Status 404 returned error can't find the container with id 422990c1f5789313630f2394c20a7c8f5f9b2a72353c225f6961dac6df8375a8 Sep 16 14:09:00.103145 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.103109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.132529 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.132495 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:09:00.132737 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.132716 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" podUID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" containerName="payload-processing" containerID="cri-o://ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0" gracePeriod=30 Sep 16 14:09:00.179669 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.179634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:00.349145 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.349114 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw"] Sep 16 14:09:00.356630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.356565 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-740a25-l4ktw"] Sep 16 14:09:00.384546 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.384523 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:09:00.481814 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.481782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvb4l\" (UniqueName: \"kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l\") pod \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " Sep 16 14:09:00.482043 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.481852 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume\") pod \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\" (UID: \"cc9bc507-b2e6-4ce0-92bd-0613672e8b88\") " Sep 16 14:09:00.482215 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.482189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc9bc507-b2e6-4ce0-92bd-0613672e8b88" (UID: "cc9bc507-b2e6-4ce0-92bd-0613672e8b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:00.483863 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.483840 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l" (OuterVolumeSpecName: "kube-api-access-xvb4l") pod "cc9bc507-b2e6-4ce0-92bd-0613672e8b88" (UID: "cc9bc507-b2e6-4ce0-92bd-0613672e8b88"). InnerVolumeSpecName "kube-api-access-xvb4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:00.583006 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.582969 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvb4l\" (UniqueName: \"kubernetes.io/projected/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-kube-api-access-xvb4l\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.583006 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.583005 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9bc507-b2e6-4ce0-92bd-0613672e8b88-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.636739 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.636649 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" event={"ID":"27ec3ce4-0b08-4f31-be7c-105b26e9a830","Type":"ContainerStarted","Data":"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4"} Sep 16 14:09:00.636739 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.636695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" event={"ID":"27ec3ce4-0b08-4f31-be7c-105b26e9a830","Type":"ContainerStarted","Data":"422990c1f5789313630f2394c20a7c8f5f9b2a72353c225f6961dac6df8375a8"} Sep 16 14:09:00.638011 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.637985 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" containerID="ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0" exitCode=0 Sep 16 14:09:00.638116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.638018 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" Sep 16 14:09:00.638116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.638037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" event={"ID":"cc9bc507-b2e6-4ce0-92bd-0613672e8b88","Type":"ContainerDied","Data":"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0"} Sep 16 14:09:00.638116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.638074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l" event={"ID":"cc9bc507-b2e6-4ce0-92bd-0613672e8b88","Type":"ContainerDied","Data":"e346ffa71cd86c32f228f981b5ad6050c01a05cb0ec0f043a68e7f0e55fbddd2"} Sep 16 14:09:00.638116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.638094 2559 scope.go:117] "RemoveContainer" containerID="ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0" Sep 16 14:09:00.647136 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.647118 2559 scope.go:117] "RemoveContainer" containerID="ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0" Sep 16 14:09:00.647404 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:00.647387 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0\": container with ID starting with ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0 not found: ID does not exist" containerID="ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0" Sep 16 14:09:00.647469 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.647417 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0"} err="failed to get container status \"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0\": rpc error: code = NotFound desc = could not find container \"ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0\": container with ID starting with ea1aaad2528c77fb506ac721e9ab522008f67860874966598505c9a1e1bd9ea0 not found: ID does not exist" Sep 16 14:09:00.664264 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.664218 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" podStartSLOduration=1.664203189 podStartE2EDuration="1.664203189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:00.662138328 +0000 UTC m=+1559.702732282" watchObservedRunningTime="2026-09-16 14:09:00.664203189 +0000 UTC m=+1559.704797142" Sep 16 14:09:00.682895 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.682851 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:09:00.693093 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:00.693067 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-740a25-94d5b6ddc-fwh4l"] Sep 16 14:09:01.138429 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.138395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:01.239096 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.239060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:01.471954 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.471923 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937cc127-d932-4d23-a793-971838700215" path="/var/lib/kubelet/pods/937cc127-d932-4d23-a793-971838700215/volumes" Sep 16 14:09:01.472312 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.472299 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" path="/var/lib/kubelet/pods/cc9bc507-b2e6-4ce0-92bd-0613672e8b88/volumes" Sep 16 14:09:01.709049 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.709014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:01.816635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:01.816546 2559 scope.go:117] "RemoveContainer" containerID="c923ef3232456b33a3b7873b331498bb40b127f03e0a5a6cfbce3530f50c8dec" Sep 16 14:09:02.841132 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:02.841097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:03.383878 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:03.383824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:09:04.976135 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:04.976099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.073470 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.073438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.173261 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.173230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.343459 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.343383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:07.685394 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.685300 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28"] Sep 16 14:09:07.685787 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.685768 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" containerName="payload-processing" Sep 16 14:09:07.685921 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.685790 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" containerName="payload-processing" Sep 16 14:09:07.685921 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.685910 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9bc507-b2e6-4ce0-92bd-0613672e8b88" containerName="payload-processing" Sep 16 14:09:07.688912 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.688891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:07.692234 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.692209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:09:07.700047 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.699982 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:09:07.703023 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.702999 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28"] Sep 16 14:09:07.746224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.746193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28\" (UID: \"9c2e5b6e-9372-42b4-be9d-b38fac2656d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:07.847446 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.847410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28\" (UID: \"9c2e5b6e-9372-42b4-be9d-b38fac2656d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:07.848048 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:07.848029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28\" (UID: \"9c2e5b6e-9372-42b4-be9d-b38fac2656d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:08.001559 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.001523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:08.273138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.273114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28"] Sep 16 14:09:08.275084 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:09:08.275053 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2e5b6e_9372_42b4_be9d_b38fac2656d7.slice/crio-08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf WatchSource:0}: Error finding container 08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf: Status 404 returned error can't find the container with id 08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf Sep 16 14:09:08.436111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.436075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:08.679664 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.679581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" event={"ID":"9c2e5b6e-9372-42b4-be9d-b38fac2656d7","Type":"ContainerStarted","Data":"df3079622560551a5c4f0694b9c784db56f412d1b90d3509948b6655caa8f2ad"} Sep 16 14:09:08.679664 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.679630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" event={"ID":"9c2e5b6e-9372-42b4-be9d-b38fac2656d7","Type":"ContainerStarted","Data":"08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf"} Sep 16 14:09:08.697075 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:08.697020 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" podStartSLOduration=1.6970032050000001 podStartE2EDuration="1.697003205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:08.695713238 +0000 UTC m=+1567.736307173" watchObservedRunningTime="2026-09-16 14:09:08.697003205 +0000 UTC m=+1567.737597159" Sep 16 14:09:09.283493 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.283408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:09.353228 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.353189 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:09.353228 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.353235 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:09.365437 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.365409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:09.686195 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.686112 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c2e5b6e-9372-42b4-be9d-b38fac2656d7" containerID="df3079622560551a5c4f0694b9c784db56f412d1b90d3509948b6655caa8f2ad" exitCode=0 Sep 16 14:09:09.686352 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.686190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" event={"ID":"9c2e5b6e-9372-42b4-be9d-b38fac2656d7","Type":"ContainerDied","Data":"df3079622560551a5c4f0694b9c784db56f412d1b90d3509948b6655caa8f2ad"} Sep 16 14:09:09.697514 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:09.697481 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:10.230685 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.230645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:10.822318 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.822295 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:10.878112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.878083 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca\") pod \"9c2e5b6e-9372-42b4-be9d-b38fac2656d7\" (UID: \"9c2e5b6e-9372-42b4-be9d-b38fac2656d7\") " Sep 16 14:09:10.878391 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.878368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c2e5b6e-9372-42b4-be9d-b38fac2656d7" (UID: "9c2e5b6e-9372-42b4-be9d-b38fac2656d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:10.878471 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.878459 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2e5b6e-9372-42b4-be9d-b38fac2656d7-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:10.883003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:10.882976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:11.697714 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:11.697678 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" event={"ID":"9c2e5b6e-9372-42b4-be9d-b38fac2656d7","Type":"ContainerDied","Data":"08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf"} Sep 16 14:09:11.697887 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:11.697725 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f377d8227d1c5da104052f92c9c7ee5eba75cd7684227c9fc6708d38f86ebf" Sep 16 14:09:11.697887 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:11.697685 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28" Sep 16 14:09:18.403968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.403931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.433400 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.433366 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:09:18.433648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.433590 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" podUID="176046fb-eff3-486c-8ed0-7bc51e3b2184" containerName="payload-processing" containerID="cri-o://c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3" gracePeriod=30 Sep 16 14:09:18.443555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.443472 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:09:18.444088 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.444040 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" podUID="6010bdfc-489d-4379-816a-de3b6a2a441c" containerName="payload-pre-processing" containerID="cri-o://bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c" gracePeriod=30 Sep 16 14:09:18.504949 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.504909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.604378 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.604346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.633218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.633185 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28"] Sep 16 14:09:18.640218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.640188 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9187f5-7kw28"] Sep 16 14:09:18.706968 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.706938 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.712543 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.712525 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:09:18.715543 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.715522 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:09:18.735213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.735189 2559 generic.go:356] "Generic (PLEG): container finished" podID="6010bdfc-489d-4379-816a-de3b6a2a441c" containerID="bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c" exitCode=0 Sep 16 14:09:18.735341 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.735220 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" Sep 16 14:09:18.735341 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.735263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" event={"ID":"6010bdfc-489d-4379-816a-de3b6a2a441c","Type":"ContainerDied","Data":"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c"} Sep 16 14:09:18.735341 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.735314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq" event={"ID":"6010bdfc-489d-4379-816a-de3b6a2a441c","Type":"ContainerDied","Data":"347946a1e093ee6b795dd2cc59eda46c040ed12bd1a0588d3327201373ca8724"} Sep 16 14:09:18.735505 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.735345 2559 scope.go:117] "RemoveContainer" containerID="bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c" Sep 16 14:09:18.736766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.736748 2559 generic.go:356] "Generic (PLEG): container finished" podID="176046fb-eff3-486c-8ed0-7bc51e3b2184" containerID="c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3" exitCode=0 Sep 16 14:09:18.736890 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.736786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" event={"ID":"176046fb-eff3-486c-8ed0-7bc51e3b2184","Type":"ContainerDied","Data":"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3"} Sep 16 14:09:18.736890 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.736797 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" Sep 16 14:09:18.736890 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.736821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88" event={"ID":"176046fb-eff3-486c-8ed0-7bc51e3b2184","Type":"ContainerDied","Data":"d7ed6dc614694e43aafbd6dfec92a0aa12523cc4496dec7adc804953c456ca31"} Sep 16 14:09:18.743878 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.743837 2559 scope.go:117] "RemoveContainer" containerID="bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c" Sep 16 14:09:18.744183 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:18.744157 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c\": container with ID starting with bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c not found: ID does not exist" containerID="bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c" Sep 16 14:09:18.744244 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.744192 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c"} err="failed to get container status \"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c\": rpc error: code = NotFound desc = could not find container \"bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c\": container with ID starting with bc87087966480465cc81ca78771173cf84cd9e14345d91f21300e970cb9f302c not found: ID does not exist" Sep 16 14:09:18.744244 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.744208 2559 scope.go:117] "RemoveContainer" containerID="c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3" Sep 16 14:09:18.754721 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.754699 2559 scope.go:117] "RemoveContainer" containerID="c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3" Sep 16 14:09:18.755107 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:18.755086 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3\": container with ID starting with c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3 not found: ID does not exist" containerID="c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3" Sep 16 14:09:18.755190 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.755113 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3"} err="failed to get container status \"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3\": rpc error: code = NotFound desc = could not find container \"c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3\": container with ID starting with c49cf31d089406f04c774bec2e5d32cded9f7577a28b85f862ef8f24852e86f3 not found: ID does not exist" Sep 16 14:09:18.855173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5pmq\" (UniqueName: \"kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq\") pod \"176046fb-eff3-486c-8ed0-7bc51e3b2184\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " Sep 16 14:09:18.855316 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855251 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume\") pod \"6010bdfc-489d-4379-816a-de3b6a2a441c\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " Sep 16 14:09:18.855316 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855285 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4ch8\" (UniqueName: \"kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8\") pod \"6010bdfc-489d-4379-816a-de3b6a2a441c\" (UID: \"6010bdfc-489d-4379-816a-de3b6a2a441c\") " Sep 16 14:09:18.855414 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855317 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume\") pod \"176046fb-eff3-486c-8ed0-7bc51e3b2184\" (UID: \"176046fb-eff3-486c-8ed0-7bc51e3b2184\") " Sep 16 14:09:18.855635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6010bdfc-489d-4379-816a-de3b6a2a441c" (UID: "6010bdfc-489d-4379-816a-de3b6a2a441c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:18.855763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.855741 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "176046fb-eff3-486c-8ed0-7bc51e3b2184" (UID: "176046fb-eff3-486c-8ed0-7bc51e3b2184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:18.857303 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.857282 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8" (OuterVolumeSpecName: "kube-api-access-c4ch8") pod "6010bdfc-489d-4379-816a-de3b6a2a441c" (UID: "6010bdfc-489d-4379-816a-de3b6a2a441c"). InnerVolumeSpecName "kube-api-access-c4ch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:18.857389 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.857365 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq" (OuterVolumeSpecName: "kube-api-access-r5pmq") pod "176046fb-eff3-486c-8ed0-7bc51e3b2184" (UID: "176046fb-eff3-486c-8ed0-7bc51e3b2184"). InnerVolumeSpecName "kube-api-access-r5pmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:18.941046 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.940966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:18.957029 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.957002 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176046fb-eff3-486c-8ed0-7bc51e3b2184-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.957029 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.957026 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5pmq\" (UniqueName: \"kubernetes.io/projected/176046fb-eff3-486c-8ed0-7bc51e3b2184-kube-api-access-r5pmq\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.957222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.957036 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6010bdfc-489d-4379-816a-de3b6a2a441c-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.957222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:18.957045 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ch8\" (UniqueName: \"kubernetes.io/projected/6010bdfc-489d-4379-816a-de3b6a2a441c-kube-api-access-c4ch8\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:19.042072 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.042040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:19.066854 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.066819 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:09:19.074407 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.074380 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9187f5-67b85fbd46-zcmfq"] Sep 16 14:09:19.086709 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.086687 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:09:19.090994 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.090973 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9187f5-5ccc669d4c-whj88"] Sep 16 14:09:19.117917 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.117896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:19.472423 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.472391 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176046fb-eff3-486c-8ed0-7bc51e3b2184" path="/var/lib/kubelet/pods/176046fb-eff3-486c-8ed0-7bc51e3b2184/volumes" Sep 16 14:09:19.472802 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.472746 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6010bdfc-489d-4379-816a-de3b6a2a441c" path="/var/lib/kubelet/pods/6010bdfc-489d-4379-816a-de3b6a2a441c/volumes" Sep 16 14:09:19.473047 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.473035 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2e5b6e-9372-42b4-be9d-b38fac2656d7" path="/var/lib/kubelet/pods/9c2e5b6e-9372-42b4-be9d-b38fac2656d7/volumes" Sep 16 14:09:19.915285 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:19.915200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:20.669593 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:20.669558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:22.135398 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:22.135360 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:09:25.163502 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:25.163464 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:25.263354 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:25.263311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:25.437228 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:25.437125 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:35.650423 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:35.650385 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:09:37.489082 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-443b8e"] Sep 16 14:09:37.489575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489549 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176046fb-eff3-486c-8ed0-7bc51e3b2184" containerName="payload-processing" Sep 16 14:09:37.489575 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489572 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="176046fb-eff3-486c-8ed0-7bc51e3b2184" containerName="payload-processing" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489587 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2e5b6e-9372-42b4-be9d-b38fac2656d7" containerName="revoke-keys" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489596 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2e5b6e-9372-42b4-be9d-b38fac2656d7" containerName="revoke-keys" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489622 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6010bdfc-489d-4379-816a-de3b6a2a441c" containerName="payload-pre-processing" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489632 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6010bdfc-489d-4379-816a-de3b6a2a441c" containerName="payload-pre-processing" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489719 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6010bdfc-489d-4379-816a-de3b6a2a441c" containerName="payload-pre-processing" Sep 16 14:09:37.489732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489729 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="176046fb-eff3-486c-8ed0-7bc51e3b2184" containerName="payload-processing" Sep 16 14:09:37.489976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.489738 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2e5b6e-9372-42b4-be9d-b38fac2656d7" containerName="revoke-keys" Sep 16 14:09:37.492774 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.492757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:37.495591 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.495572 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:09:37.496170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.496147 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-443b8e"] Sep 16 14:09:37.531015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.530990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4m2\" (UniqueName: \"kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2\") pod \"test-curl-7899-443b8e\" (UID: \"8c71a236-3005-47de-861c-377bd59a52ba\") " pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:37.631973 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.631938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4m2\" (UniqueName: \"kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2\") pod \"test-curl-7899-443b8e\" (UID: \"8c71a236-3005-47de-861c-377bd59a52ba\") " pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:37.640884 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.640851 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4m2\" (UniqueName: \"kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2\") pod \"test-curl-7899-443b8e\" (UID: \"8c71a236-3005-47de-861c-377bd59a52ba\") " pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:37.803675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:37.803589 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:38.078936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:38.078909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-443b8e"] Sep 16 14:09:38.080972 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:09:38.080936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c71a236_3005_47de_861c_377bd59a52ba.slice/crio-eb2b8f3ebe68e400d0c7925c439f0b0f3547eeebcc6f960cca88b33a01a8ea5e WatchSource:0}: Error finding container eb2b8f3ebe68e400d0c7925c439f0b0f3547eeebcc6f960cca88b33a01a8ea5e: Status 404 returned error can't find the container with id eb2b8f3ebe68e400d0c7925c439f0b0f3547eeebcc6f960cca88b33a01a8ea5e Sep 16 14:09:38.839442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:38.839403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-443b8e" event={"ID":"8c71a236-3005-47de-861c-377bd59a52ba","Type":"ContainerStarted","Data":"7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284"} Sep 16 14:09:38.839442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:38.839448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-443b8e" event={"ID":"8c71a236-3005-47de-861c-377bd59a52ba","Type":"ContainerStarted","Data":"eb2b8f3ebe68e400d0c7925c439f0b0f3547eeebcc6f960cca88b33a01a8ea5e"} Sep 16 14:09:38.858034 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:38.857984 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7899-443b8e" podStartSLOduration=1.8579701549999998 podStartE2EDuration="1.857970155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:38.856363571 +0000 UTC m=+1597.896957526" watchObservedRunningTime="2026-09-16 14:09:38.857970155 +0000 UTC m=+1597.898564108" Sep 16 14:09:39.475224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.475184 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-443b8e"] Sep 16 14:09:39.480150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.480119 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-443b8e"] Sep 16 14:09:39.595685 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.595597 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-8b8575"] Sep 16 14:09:39.595982 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.595965 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c71a236-3005-47de-861c-377bd59a52ba" containerName="test-curl-7899-443b8e" Sep 16 14:09:39.595982 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.595982 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c71a236-3005-47de-861c-377bd59a52ba" containerName="test-curl-7899-443b8e" Sep 16 14:09:39.596132 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.596057 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c71a236-3005-47de-861c-377bd59a52ba" containerName="test-curl-7899-443b8e" Sep 16 14:09:39.599111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.599094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:39.603509 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.603486 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-8b8575"] Sep 16 14:09:39.647146 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.647111 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgwc\" (UniqueName: \"kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc\") pod \"test-curl-7899-8b8575\" (UID: \"2fce2394-4352-4a74-9bb7-e9af2cd92c4e\") " pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:39.748239 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.748212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgwc\" (UniqueName: \"kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc\") pod \"test-curl-7899-8b8575\" (UID: \"2fce2394-4352-4a74-9bb7-e9af2cd92c4e\") " pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:39.758021 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.758001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgwc\" (UniqueName: \"kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc\") pod \"test-curl-7899-8b8575\" (UID: \"2fce2394-4352-4a74-9bb7-e9af2cd92c4e\") " pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:39.844409 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:39.844370 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7899-443b8e" container="test-curl-7899-443b8e" Sep 16 14:09:39.909792 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:39.909717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:40.182085 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:40.182001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-8b8575"] Sep 16 14:09:40.185221 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:09:40.185193 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fce2394_4352_4a74_9bb7_e9af2cd92c4e.slice/crio-5646b69f9cd8ede728cc83a6975f07e51ad774dae5d0b010e07f6d40daa84880 WatchSource:0}: Error finding container 5646b69f9cd8ede728cc83a6975f07e51ad774dae5d0b010e07f6d40daa84880: Status 404 returned error can't find the container with id 5646b69f9cd8ede728cc83a6975f07e51ad774dae5d0b010e07f6d40daa84880 Sep 16 14:09:40.851356 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:40.851313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-8b8575" event={"ID":"2fce2394-4352-4a74-9bb7-e9af2cd92c4e","Type":"ContainerStarted","Data":"027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8"} Sep 16 14:09:40.851356 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:40.851362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-8b8575" event={"ID":"2fce2394-4352-4a74-9bb7-e9af2cd92c4e","Type":"ContainerStarted","Data":"5646b69f9cd8ede728cc83a6975f07e51ad774dae5d0b010e07f6d40daa84880"} Sep 16 14:09:40.851845 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:40.851433 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7899-443b8e" podUID="8c71a236-3005-47de-861c-377bd59a52ba" containerName="test-curl-7899-443b8e" containerID="cri-o://7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284" gracePeriod=2 Sep 16 14:09:40.853370 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:40.853343 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-443b8e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8c71a236-3005-47de-861c-377bd59a52ba" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:40.872021 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:40.871984 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-443b8e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8c71a236-3005-47de-861c-377bd59a52ba" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:40.872209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:40.872162 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7899-8b8575" podStartSLOduration=1.872144992 podStartE2EDuration="1.872144992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:40.870142921 +0000 UTC m=+1599.910736888" watchObservedRunningTime="2026-09-16 14:09:40.872144992 +0000 UTC m=+1599.912738948" Sep 16 14:09:41.470045 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:41.470011 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-443b8e\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="8c71a236-3005-47de-861c-377bd59a52ba" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:41.479985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:41.479952 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-8b8575"] Sep 16 14:09:41.488364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:41.488334 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-8b8575"] Sep 16 14:09:41.856417 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:41.856385 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7899-8b8575" container="test-curl-7899-8b8575" Sep 16 14:09:42.861236 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:42.861199 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7899-8b8575" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" containerName="test-curl-7899-8b8575" containerID="cri-o://027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8" gracePeriod=2 Sep 16 14:09:42.863719 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:42.863686 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-8b8575\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:43.020201 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.020178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:43.022693 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:43.022670 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-8b8575\" is forbidden: User \"system:node:ip-10-0-139-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-34.ec2.internal' and this object" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:43.079483 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.079459 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf4m2\" (UniqueName: \"kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2\") pod \"8c71a236-3005-47de-861c-377bd59a52ba\" (UID: \"8c71a236-3005-47de-861c-377bd59a52ba\") " Sep 16 14:09:43.081375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.081347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2" (OuterVolumeSpecName: "kube-api-access-qf4m2") pod "8c71a236-3005-47de-861c-377bd59a52ba" (UID: "8c71a236-3005-47de-861c-377bd59a52ba"). InnerVolumeSpecName "kube-api-access-qf4m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:43.181007 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.180937 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4m2\" (UniqueName: \"kubernetes.io/projected/8c71a236-3005-47de-861c-377bd59a52ba-kube-api-access-qf4m2\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:43.471176 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.471145 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c71a236-3005-47de-861c-377bd59a52ba" path="/var/lib/kubelet/pods/8c71a236-3005-47de-861c-377bd59a52ba/volumes" Sep 16 14:09:43.522482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.522440 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:09:43.557698 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.557671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:43.867178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.867086 2559 generic.go:356] "Generic (PLEG): container finished" podID="8c71a236-3005-47de-861c-377bd59a52ba" containerID="7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284" exitCode=137 Sep 16 14:09:43.867178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.867122 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-443b8e" Sep 16 14:09:43.867625 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.867197 2559 scope.go:117] "RemoveContainer" containerID="7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284" Sep 16 14:09:43.875854 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.875811 2559 scope.go:117] "RemoveContainer" containerID="7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284" Sep 16 14:09:43.876109 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:43.876091 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284\": container with ID starting with 7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284 not found: ID does not exist" containerID="7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284" Sep 16 14:09:43.876165 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:43.876118 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284"} err="failed to get container status \"7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284\": rpc error: code = NotFound desc = could not find container \"7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284\": container with ID starting with 7fa9e04b61c8d9d7a63962a581f57382c685185d7ba2003502c2e4608d1b9284 not found: ID does not exist" Sep 16 14:09:44.341135 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:44.341103 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:09:44.379570 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:44.379520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:45.047846 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.047816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:45.100839 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.100805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtgwc\" (UniqueName: \"kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc\") pod \"2fce2394-4352-4a74-9bb7-e9af2cd92c4e\" (UID: \"2fce2394-4352-4a74-9bb7-e9af2cd92c4e\") " Sep 16 14:09:45.102844 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.102818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc" (OuterVolumeSpecName: "kube-api-access-mtgwc") pod "2fce2394-4352-4a74-9bb7-e9af2cd92c4e" (UID: "2fce2394-4352-4a74-9bb7-e9af2cd92c4e"). InnerVolumeSpecName "kube-api-access-mtgwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:45.202462 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.202428 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgwc\" (UniqueName: \"kubernetes.io/projected/2fce2394-4352-4a74-9bb7-e9af2cd92c4e-kube-api-access-mtgwc\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:45.471495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.471412 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" path="/var/lib/kubelet/pods/2fce2394-4352-4a74-9bb7-e9af2cd92c4e/volumes" Sep 16 14:09:45.879797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.879713 2559 generic.go:356] "Generic (PLEG): container finished" podID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" containerID="027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8" exitCode=137 Sep 16 14:09:45.879797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.879754 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-8b8575" Sep 16 14:09:45.880048 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.879812 2559 scope.go:117] "RemoveContainer" containerID="027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8" Sep 16 14:09:45.888245 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.888231 2559 scope.go:117] "RemoveContainer" containerID="027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8" Sep 16 14:09:45.888518 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:45.888499 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8\": container with ID starting with 027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8 not found: ID does not exist" containerID="027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8" Sep 16 14:09:45.888556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:45.888529 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8"} err="failed to get container status \"027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8\": rpc error: code = NotFound desc = could not find container \"027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8\": container with ID starting with 027c1c7bc73ed1fae596208e49cd5d736524674a798c9640b7f0068a23c3b0b8 not found: ID does not exist" Sep 16 14:09:46.656242 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:46.656206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:09:54.652481 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:54.652404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:09:58.138856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.138821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:09:58.540630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.540579 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" podUID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" containerName="main" containerID="cri-o://55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e" gracePeriod=45 Sep 16 14:09:58.797162 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.797078 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.933967 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934042 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934086 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fdpb\" (UniqueName: \"kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934172 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934204 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934245 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home\") pod \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\" (UID: \"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6\") " Sep 16 14:09:58.934990 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.934692 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home" (OuterVolumeSpecName: "home") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:58.936177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.935717 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:58.936177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.935886 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:58.941535 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.941482 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm" (OuterVolumeSpecName: "dshm") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:58.943547 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.943510 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:58.945650 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.945330 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb" (OuterVolumeSpecName: "kube-api-access-5fdpb") pod "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" (UID: "29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6"). InnerVolumeSpecName "kube-api-access-5fdpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:58.950384 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.950351 2559 generic.go:356] "Generic (PLEG): container finished" podID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" containerID="55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e" exitCode=0 Sep 16 14:09:58.950507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.950391 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" Sep 16 14:09:58.950507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.950395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" event={"ID":"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6","Type":"ContainerDied","Data":"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e"} Sep 16 14:09:58.950507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.950447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf" event={"ID":"29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6","Type":"ContainerDied","Data":"26e9b30c7ad76a737cc822948bf2b24eba0188dd0343314c5a3897456b84af9d"} Sep 16 14:09:58.950507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.950475 2559 scope.go:117] "RemoveContainer" containerID="55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e" Sep 16 14:09:58.963048 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.963019 2559 scope.go:117] "RemoveContainer" containerID="55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e" Sep 16 14:09:58.963406 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:58.963375 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e\": container with ID starting with 55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e not found: ID does not exist" containerID="55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e" Sep 16 14:09:58.963497 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.963418 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e"} err="failed to get container status \"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e\": rpc error: code = NotFound desc = could not find container \"55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e\": container with ID starting with 55a8981812ca705efe815cdae9087cf1313a6bb46e9d9ac4e69888baf79eb89e not found: ID does not exist" Sep 16 14:09:58.975257 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.975217 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:09:58.981904 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:58.981859 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/sub-test-model-46224d-kserve-5dd8d9c8b7-4bnmf"] Sep 16 14:09:59.035747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035700 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.035747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035740 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fdpb\" (UniqueName: \"kubernetes.io/projected/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-kube-api-access-5fdpb\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.035747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035752 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.035747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035762 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.036114 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035770 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.036114 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.035777 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.359259 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.359206 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" podUID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" containerName="main" containerID="cri-o://16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4" gracePeriod=45 Sep 16 14:09:59.471677 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.471651 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" path="/var/lib/kubelet/pods/29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6/volumes" Sep 16 14:09:59.593071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.593048 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:59.742701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742664 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.742929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742711 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g4v6\" (UniqueName: \"kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.742929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.742929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742799 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.742929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742819 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.742929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.742904 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir\") pod \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\" (UID: \"27ec3ce4-0b08-4f31-be7c-105b26e9a830\") " Sep 16 14:09:59.743191 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.743016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home" (OuterVolumeSpecName: "home") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:59.743191 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.743144 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.743191 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.743175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache" (OuterVolumeSpecName: "model-cache") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:59.743332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.743214 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:59.745014 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.744991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:59.745014 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.745007 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6" (OuterVolumeSpecName: "kube-api-access-2g4v6") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "kube-api-access-2g4v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:59.745153 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.744997 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm" (OuterVolumeSpecName: "dshm") pod "27ec3ce4-0b08-4f31-be7c-105b26e9a830" (UID: "27ec3ce4-0b08-4f31-be7c-105b26e9a830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:09:59.843936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.843865 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.843936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.843929 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4v6\" (UniqueName: \"kubernetes.io/projected/27ec3ce4-0b08-4f31-be7c-105b26e9a830-kube-api-access-2g4v6\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.843936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.843939 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ec3ce4-0b08-4f31-be7c-105b26e9a830-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.843936 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.843948 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.844205 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.843957 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ec3ce4-0b08-4f31-be7c-105b26e9a830-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:09:59.957071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.957038 2559 generic.go:356] "Generic (PLEG): container finished" podID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" containerID="16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4" exitCode=0 Sep 16 14:09:59.957071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.957078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" event={"ID":"27ec3ce4-0b08-4f31-be7c-105b26e9a830","Type":"ContainerDied","Data":"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4"} Sep 16 14:09:59.957286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.957083 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" Sep 16 14:09:59.957286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.957103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb" event={"ID":"27ec3ce4-0b08-4f31-be7c-105b26e9a830","Type":"ContainerDied","Data":"422990c1f5789313630f2394c20a7c8f5f9b2a72353c225f6961dac6df8375a8"} Sep 16 14:09:59.957286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.957119 2559 scope.go:117] "RemoveContainer" containerID="16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4" Sep 16 14:09:59.966754 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.966736 2559 scope.go:117] "RemoveContainer" containerID="16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4" Sep 16 14:09:59.967043 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:09:59.967025 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4\": container with ID starting with 16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4 not found: ID does not exist" containerID="16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4" Sep 16 14:09:59.967102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.967051 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4"} err="failed to get container status \"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4\": rpc error: code = NotFound desc = could not find container \"16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4\": container with ID starting with 16804b3f5c8ee3af756c29ef0ebce5a81d840b842ffdbf9b5dbc6607b1c015e4 not found: ID does not exist" Sep 16 14:09:59.981910 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.981884 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:09:59.986703 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:09:59.986681 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/sub-test-model-b4884e-kserve-69c95c69fc-9gnbb"] Sep 16 14:10:01.470883 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:01.470836 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" path="/var/lib/kubelet/pods/27ec3ce4-0b08-4f31-be7c-105b26e9a830/volumes" Sep 16 14:10:27.252002 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.251966 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:10:27.252537 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252518 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" containerName="main" Sep 16 14:10:27.252581 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252541 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" containerName="main" Sep 16 14:10:27.252581 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252572 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" containerName="main" Sep 16 14:10:27.252649 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252581 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" containerName="main" Sep 16 14:10:27.252649 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252605 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" containerName="test-curl-7899-8b8575" Sep 16 14:10:27.252649 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252614 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" containerName="test-curl-7899-8b8575" Sep 16 14:10:27.252738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252718 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c91edd-6bce-4a8e-ba09-2fb57f0ae4f6" containerName="main" Sep 16 14:10:27.252738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252732 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fce2394-4352-4a74-9bb7-e9af2cd92c4e" containerName="test-curl-7899-8b8575" Sep 16 14:10:27.252809 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.252745 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ec3ce4-0b08-4f31-be7c-105b26e9a830" containerName="main" Sep 16 14:10:27.256154 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.256137 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.258544 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.258521 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"default-dockercfg-7k44r\"" Sep 16 14:10:27.259906 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.259886 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"rate-test-model-b4884e-kserve-self-signed-certs\"" Sep 16 14:10:27.259986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.259886 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"kube-root-ca.crt\"" Sep 16 14:10:27.259986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.259925 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b4884e\"/\"openshift-service-ca.crt\"" Sep 16 14:10:27.263787 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.263761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:10:27.274770 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.274741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:27.399507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.399507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5l9\" (UniqueName: \"kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.399738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.399738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.399738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.399738 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.399686 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.500915 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.500861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.500930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.500964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.500993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.501046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501112 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.501068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5l9\" (UniqueName: \"kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.501236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501373 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.501357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.501492 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.501385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.503259 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.503197 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.503392 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.503376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.511929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.511904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5l9\" (UniqueName: \"kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9\") pod \"rate-test-model-b4884e-kserve-f675974f5-rgzxr\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.565637 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.565590 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:27.842033 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:27.842005 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:10:27.844366 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:10:27.844341 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3366307b_e055_461e_b804_62a0f2184a21.slice/crio-b656ab911abc0be3fe862b172c54461fcd234510744eda149cb19ab7dfbc9482 WatchSource:0}: Error finding container b656ab911abc0be3fe862b172c54461fcd234510744eda149cb19ab7dfbc9482: Status 404 returned error can't find the container with id b656ab911abc0be3fe862b172c54461fcd234510744eda149cb19ab7dfbc9482 Sep 16 14:10:28.097522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:28.097428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" event={"ID":"3366307b-e055-461e-b804-62a0f2184a21","Type":"ContainerStarted","Data":"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b"} Sep 16 14:10:28.097522 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:28.097473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" event={"ID":"3366307b-e055-461e-b804-62a0f2184a21","Type":"ContainerStarted","Data":"b656ab911abc0be3fe862b172c54461fcd234510744eda149cb19ab7dfbc9482"} Sep 16 14:10:28.115302 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:28.115249 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" podStartSLOduration=1.115234267 podStartE2EDuration="1.115234267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:10:28.114126716 +0000 UTC m=+1647.154720671" watchObservedRunningTime="2026-09-16 14:10:28.115234267 +0000 UTC m=+1647.155828240" Sep 16 14:10:30.846806 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:30.846763 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:10:37.566638 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:37.566597 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:37.567156 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:37.566789 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:37.578459 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:37.578436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:38.156027 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:38.155995 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:10:47.245213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:47.245172 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:10:56.641491 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:10:56.641455 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:11:03.414469 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:03.414434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:04.370063 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:04.370028 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:11:04.385093 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:04.385059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:04.829766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:04.829714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:05.136246 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.136169 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:11:05.138453 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.138432 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.140699 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.140679 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cabdc2-openshift-default-dockercfg-jpvf6\"" Sep 16 14:11:05.159931 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.159907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:11:05.245236 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.245202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:05.254456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254532 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254532 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ngv9\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254670 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254670 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254636 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254670 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.254760 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.254717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355628 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355628 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ngv9\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.355831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.355831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.356232 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.356174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.356232 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.356214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.356352 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.356333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.356396 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.356365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.356501 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.356478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.357993 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.357974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.358132 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.358114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.364170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.364148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.364251 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.364211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ngv9\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9\") pod \"e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.448545 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.448513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:05.723764 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.723588 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:11:05.726032 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:11:05.725987 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775682c6_f582_455c_bbf2_b5c2244b7cb4.slice/crio-724a8b750cd1f4f3a7a8f247265f0cdcf9d66121dc52fcdb643de296b5129308 WatchSource:0}: Error finding container 724a8b750cd1f4f3a7a8f247265f0cdcf9d66121dc52fcdb643de296b5129308: Status 404 returned error can't find the container with id 724a8b750cd1f4f3a7a8f247265f0cdcf9d66121dc52fcdb643de296b5129308 Sep 16 14:11:05.728711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.728679 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:05.728797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.728746 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:05.728797 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.728774 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:05.809179 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:05.809092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:06.278433 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:06.278398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerStarted","Data":"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97"} Sep 16 14:11:06.278805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:06.278446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerStarted","Data":"724a8b750cd1f4f3a7a8f247265f0cdcf9d66121dc52fcdb643de296b5129308"} Sep 16 14:11:06.306157 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:06.306108 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podStartSLOduration=1.306093439 podStartE2EDuration="1.306093439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:06.304023662 +0000 UTC m=+1685.344617619" watchObservedRunningTime="2026-09-16 14:11:06.306093439 +0000 UTC m=+1685.346687392" Sep 16 14:11:06.448703 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:06.448664 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:06.453710 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:06.453682 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:07.045302 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:07.045268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:11:07.284367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:07.284334 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:07.285120 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:07.285100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:07.592292 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:07.592257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:08.836978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:08.836939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:08.940264 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:08.940230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:09.536287 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:09.536249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:10.377043 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:10.377010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:10.443102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:10.443070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:10.843524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:10.843489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:11.018925 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.018858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:11.118104 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.118015 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:11.200935 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.200895 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:11:11.203304 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.203287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.206022 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.205998 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cabdc2-dockercfg-9bb7v\"" Sep 16 14:11:11.206170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.205998 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cabdc2\"" Sep 16 14:11:11.212631 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.212604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:11:11.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.314105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zz2\" (UniqueName: \"kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.314143 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.314144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.415833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.415746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zz2\" (UniqueName: \"kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.415833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.415795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.416479 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.416458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.425106 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.425076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zz2\" (UniqueName: \"kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2\") pod \"payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.514762 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.514727 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:11.786342 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:11.786315 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:11:11.788370 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:11:11.788340 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bfb05c_2679_458e_8e68_31c658ab7a91.slice/crio-ab4e70662c31efc33569eab7bf6fd6e2842b82756a7c1dfe6daa26f16b45f0c1 WatchSource:0}: Error finding container ab4e70662c31efc33569eab7bf6fd6e2842b82756a7c1dfe6daa26f16b45f0c1: Status 404 returned error can't find the container with id ab4e70662c31efc33569eab7bf6fd6e2842b82756a7c1dfe6daa26f16b45f0c1 Sep 16 14:11:12.310263 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:12.310225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" event={"ID":"d1bfb05c-2679-458e-8e68-31c658ab7a91","Type":"ContainerStarted","Data":"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808"} Sep 16 14:11:12.310460 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:12.310288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" event={"ID":"d1bfb05c-2679-458e-8e68-31c658ab7a91","Type":"ContainerStarted","Data":"ab4e70662c31efc33569eab7bf6fd6e2842b82756a7c1dfe6daa26f16b45f0c1"} Sep 16 14:11:12.310460 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:12.310351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:12.328217 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:12.328163 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" podStartSLOduration=1.328147717 podStartE2EDuration="1.328147717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:12.326916711 +0000 UTC m=+1691.367510666" watchObservedRunningTime="2026-09-16 14:11:12.328147717 +0000 UTC m=+1691.368741671" Sep 16 14:11:19.388368 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.388326 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" podUID="3366307b-e055-461e-b804-62a0f2184a21" containerName="main" containerID="cri-o://65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b" gracePeriod=45 Sep 16 14:11:19.678807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.678738 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:11:19.694138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694106 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x5l9\" (UniqueName: \"kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694207 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694315 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm\") pod \"3366307b-e055-461e-b804-62a0f2184a21\" (UID: \"3366307b-e055-461e-b804-62a0f2184a21\") " Sep 16 14:11:19.694565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694498 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:19.694565 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694522 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache" (OuterVolumeSpecName: "model-cache") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:19.694645 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.694579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home" (OuterVolumeSpecName: "home") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:19.696290 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.696261 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm" (OuterVolumeSpecName: "dshm") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:19.696417 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.696380 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9" (OuterVolumeSpecName: "kube-api-access-8x5l9") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "kube-api-access-8x5l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:11:19.696417 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.696391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3366307b-e055-461e-b804-62a0f2184a21" (UID: "3366307b-e055-461e-b804-62a0f2184a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:19.794934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794894 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3366307b-e055-461e-b804-62a0f2184a21-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:19.794934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794931 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:19.794934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794944 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x5l9\" (UniqueName: \"kubernetes.io/projected/3366307b-e055-461e-b804-62a0f2184a21-kube-api-access-8x5l9\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:19.794934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794956 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:19.795229 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794969 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:19.795229 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:19.794980 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3366307b-e055-461e-b804-62a0f2184a21-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:11:20.042537 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.042498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:11:20.351938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.351818 2559 generic.go:356] "Generic (PLEG): container finished" podID="3366307b-e055-461e-b804-62a0f2184a21" containerID="65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b" exitCode=0 Sep 16 14:11:20.351938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.351903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" event={"ID":"3366307b-e055-461e-b804-62a0f2184a21","Type":"ContainerDied","Data":"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b"} Sep 16 14:11:20.352182 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.351944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" event={"ID":"3366307b-e055-461e-b804-62a0f2184a21","Type":"ContainerDied","Data":"b656ab911abc0be3fe862b172c54461fcd234510744eda149cb19ab7dfbc9482"} Sep 16 14:11:20.352182 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.351970 2559 scope.go:117] "RemoveContainer" containerID="65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b" Sep 16 14:11:20.352182 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.352149 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr" Sep 16 14:11:20.380593 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.380567 2559 scope.go:117] "RemoveContainer" containerID="65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b" Sep 16 14:11:20.380961 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:11:20.380937 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b\": container with ID starting with 65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b not found: ID does not exist" containerID="65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b" Sep 16 14:11:20.381045 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.380973 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b"} err="failed to get container status \"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b\": rpc error: code = NotFound desc = could not find container \"65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b\": container with ID starting with 65db370e7eb43ac93c7f80fab180234122d55291dbab12ac086882c3562dd90b not found: ID does not exist" Sep 16 14:11:20.392367 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.392336 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:11:20.396568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:20.396544 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b4884e/rate-test-model-b4884e-kserve-f675974f5-rgzxr"] Sep 16 14:11:21.472547 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:21.472507 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3366307b-e055-461e-b804-62a0f2184a21" path="/var/lib/kubelet/pods/3366307b-e055-461e-b804-62a0f2184a21/volumes" Sep 16 14:11:23.318003 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:23.317966 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:11:23.327024 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:23.326992 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 14:11:23.327204 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:23.327060 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 14:11:24.374996 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.374911 2559 generic.go:356] "Generic (PLEG): container finished" podID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerID="e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97" exitCode=0 Sep 16 14:11:24.375416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.374985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerDied","Data":"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97"} Sep 16 14:11:24.375416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.375376 2559 scope.go:117] "RemoveContainer" containerID="e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97" Sep 16 14:11:24.376508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.376466 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:24.376566 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.376556 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:24.376603 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:24.376587 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:25.381986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:25.381943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerStarted","Data":"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3"} Sep 16 14:11:25.448958 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:25.448847 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:25.449237 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:25.449208 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Sep 16 14:11:25.449316 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:25.449276 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Sep 16 14:11:26.449660 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:26.449618 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Sep 16 14:11:26.450349 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:26.450312 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Sep 16 14:11:27.450073 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:27.450027 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Sep 16 14:11:27.450564 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:27.450104 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Sep 16 14:11:28.449719 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:28.449670 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Sep 16 14:11:28.449965 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:28.449754 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Sep 16 14:11:29.449939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:29.449889 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Sep 16 14:11:29.450445 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:29.449964 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Sep 16 14:11:30.453380 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:30.453342 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:30.453850 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:30.453718 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:30.454484 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:30.454464 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:11:32.885311 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:32.885272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:33.247103 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:33.247063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:33.939249 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:33.939205 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:34.028240 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.028195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:34.237148 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.237114 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:11:34.237517 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.237501 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3366307b-e055-461e-b804-62a0f2184a21" containerName="main" Sep 16 14:11:34.237585 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.237520 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3366307b-e055-461e-b804-62a0f2184a21" containerName="main" Sep 16 14:11:34.237678 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.237593 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3366307b-e055-461e-b804-62a0f2184a21" containerName="main" Sep 16 14:11:34.239704 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.239683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.242279 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.242257 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9c8daa-openshift-default-dockercfg-h9f5c\"" Sep 16 14:11:34.257560 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.257534 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:11:34.273415 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.273385 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:34.335009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.334972 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335189 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335186 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335390 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335222 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6tr\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335390 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.335390 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.335260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.436763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.436763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.436972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437095 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6tr\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437381 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437428 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.437951 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.437926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.439355 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.439329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.439504 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.439486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.449634 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.449602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.449892 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.449849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6tr\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr\") pod \"e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.550132 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.550035 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:34.707170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.707135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:34.830973 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.830946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:11:34.832755 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:11:34.832723 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c985712_a57e_4ee4_8269_604d564c1e5b.slice/crio-bd7ddc22404bdec4fcb0adbbd02b4b6dccb80702e89e5e42674073a80b35a66b WatchSource:0}: Error finding container bd7ddc22404bdec4fcb0adbbd02b4b6dccb80702e89e5e42674073a80b35a66b: Status 404 returned error can't find the container with id bd7ddc22404bdec4fcb0adbbd02b4b6dccb80702e89e5e42674073a80b35a66b Sep 16 14:11:34.834849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.834816 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:34.834981 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.834913 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:34.834981 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.834960 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:34.897503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:34.897475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:35.434103 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.434066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerStarted","Data":"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66"} Sep 16 14:11:35.434490 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.434141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerStarted","Data":"bd7ddc22404bdec4fcb0adbbd02b4b6dccb80702e89e5e42674073a80b35a66b"} Sep 16 14:11:35.458291 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.458229 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podStartSLOduration=1.4582135090000001 podStartE2EDuration="1.458213509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:35.455112632 +0000 UTC m=+1714.495706588" watchObservedRunningTime="2026-09-16 14:11:35.458213509 +0000 UTC m=+1714.498807463" Sep 16 14:11:35.550324 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.550276 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:35.555433 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.555405 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:35.769482 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:35.769435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:36.439403 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:36.439366 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:36.440364 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:36.440333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:36.757561 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:36.757523 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:38.150181 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:38.150121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:38.250106 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:38.250055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:38.669932 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:38.669861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:39.419304 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:39.419214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:40.322302 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.322269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:40.421455 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.421423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:40.790960 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.790919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:40.889382 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.889336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:40.919402 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.919367 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:11:40.921953 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.921932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:40.924268 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.924246 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9c8daa-dockercfg-9qwfw\"" Sep 16 14:11:40.924698 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.924681 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9c8daa\"" Sep 16 14:11:40.935543 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:40.935516 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:11:41.011722 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.011675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.011722 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.011729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcmh\" (UniqueName: \"kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.112772 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.112677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.112772 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.112729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcmh\" (UniqueName: \"kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.113460 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.113433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.121767 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.121742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcmh\" (UniqueName: \"kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh\") pod \"payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.233066 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.233028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:41.328214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.328178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:41.503583 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:41.503555 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:11:41.506117 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:11:41.506088 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d055879_6bb3_47a8_9c6f_eb61b9f113a0.slice/crio-0fb2499fe0fdc4c2d4ec8af89f271a469175f65d0686e022ceeae8199ca91c3a WatchSource:0}: Error finding container 0fb2499fe0fdc4c2d4ec8af89f271a469175f65d0686e022ceeae8199ca91c3a: Status 404 returned error can't find the container with id 0fb2499fe0fdc4c2d4ec8af89f271a469175f65d0686e022ceeae8199ca91c3a Sep 16 14:11:42.473711 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:42.473671 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" event={"ID":"2d055879-6bb3-47a8-9c6f-eb61b9f113a0","Type":"ContainerStarted","Data":"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059"} Sep 16 14:11:42.473937 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:42.473725 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" event={"ID":"2d055879-6bb3-47a8-9c6f-eb61b9f113a0","Type":"ContainerStarted","Data":"0fb2499fe0fdc4c2d4ec8af89f271a469175f65d0686e022ceeae8199ca91c3a"} Sep 16 14:11:42.473937 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:42.473747 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:42.493488 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:42.493426 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" podStartSLOduration=2.493408297 podStartE2EDuration="2.493408297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:42.49115709 +0000 UTC m=+1721.531751045" watchObservedRunningTime="2026-09-16 14:11:42.493408297 +0000 UTC m=+1721.534002289" Sep 16 14:11:44.948153 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:44.948119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:11:53.479622 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:53.479581 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:11:55.544662 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.544628 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerID="8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66" exitCode=0 Sep 16 14:11:55.545104 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.544700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerDied","Data":"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66"} Sep 16 14:11:55.545165 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.545116 2559 scope.go:117] "RemoveContainer" containerID="8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66" Sep 16 14:11:55.546206 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.546177 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:55.546274 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.546258 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:55.546317 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:55.546290 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 14:11:56.554824 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:56.554779 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerStarted","Data":"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49"} Sep 16 14:11:57.551099 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:57.551053 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:11:57.551366 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:57.551337 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:11:57.551440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:57.551402 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 16 14:11:58.551328 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:58.551282 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:11:58.551849 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:58.551361 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 16 14:11:59.550782 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:59.550743 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:11:59.551106 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:11:59.551064 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 16 14:12:00.551149 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:00.551112 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:12:00.551777 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:00.551725 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 16 14:12:01.551180 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:01.551135 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 16 14:12:01.551786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:01.551753 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 16 14:12:02.392584 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:02.392544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:02.495412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:02.495372 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:02.555465 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:02.555435 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:12:02.555927 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:02.555827 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:12:02.556531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:02.556515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:12:03.386976 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:03.386944 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:04.147357 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:04.147325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:05.222732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:05.222688 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:07.195861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:07.195818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:14.329208 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.329175 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:12:14.332806 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.332789 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.335641 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.335591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cabdc2\"/\"kube-root-ca.crt\"" Sep 16 14:12:14.336541 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.336522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cabdc2\"/\"ipp-route-cabdc2-kserve-self-signed-certs\"" Sep 16 14:12:14.336741 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.336726 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cabdc2\"/\"openshift-service-ca.crt\"" Sep 16 14:12:14.336833 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.336813 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cabdc2\"/\"default-dockercfg-pkdz2\"" Sep 16 14:12:14.342630 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.342604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:12:14.364403 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.364364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:14.445439 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445401 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.445439 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddvh\" (UniqueName: \"kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.445682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.445682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.445758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.445758 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.445724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547268 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547268 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547506 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547506 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddvh\" (UniqueName: \"kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547602 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547602 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547725 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547791 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.547791 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.547769 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.549788 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.549758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.550139 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.550120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.556574 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.556547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddvh\" (UniqueName: \"kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh\") pod \"ipp-route-cabdc2-kserve-7f4fbd5655-kf6km\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.643414 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.643317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:14.916150 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:14.916064 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:12:14.919282 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:12:14.919256 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f2efa78_b160_4f76_856d_9314fb222f8b.slice/crio-4a2a9b72d1729ffcc8193ec5761f7917aa1a9180e9aefd1e318a0a678a32daad WatchSource:0}: Error finding container 4a2a9b72d1729ffcc8193ec5761f7917aa1a9180e9aefd1e318a0a678a32daad: Status 404 returned error can't find the container with id 4a2a9b72d1729ffcc8193ec5761f7917aa1a9180e9aefd1e318a0a678a32daad Sep 16 14:12:15.660420 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:15.660379 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" event={"ID":"7f2efa78-b160-4f76-856d-9314fb222f8b","Type":"ContainerStarted","Data":"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c"} Sep 16 14:12:15.660420 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:15.660423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" event={"ID":"7f2efa78-b160-4f76-856d-9314fb222f8b","Type":"ContainerStarted","Data":"4a2a9b72d1729ffcc8193ec5761f7917aa1a9180e9aefd1e318a0a678a32daad"} Sep 16 14:12:15.680213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:15.680159 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" podStartSLOduration=1.6801413699999999 podStartE2EDuration="1.68014137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:15.678384765 +0000 UTC m=+1754.718978715" watchObservedRunningTime="2026-09-16 14:12:15.68014137 +0000 UTC m=+1754.720735324" Sep 16 14:12:18.242261 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:18.242220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:12:24.643999 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:24.643955 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:24.644432 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:24.644014 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:24.656721 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:24.656686 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:24.714412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:24.714375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:12:32.143668 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:32.143626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:12:37.175997 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:37.175957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:38.903732 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:38.903692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:39.249494 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:39.249451 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:39.549896 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:39.549793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:41.409486 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:41.409445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:42.095760 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:42.095724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:42.197857 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:42.197818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:42.477160 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:42.477123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:43.269218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:43.269174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:44.181286 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:44.181249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:44.975525 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:44.975485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:45.699001 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:45.698967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:45.776454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:45.776406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:45.931030 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:45.930993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:46.209809 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.209772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:46.310124 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.310085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:46.325929 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.325890 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:12:46.336655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.336621 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.339456 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.339422 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-29812a\"" Sep 16 14:12:46.339774 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.339458 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-29812a-dockercfg-zgcrv\"" Sep 16 14:12:46.340660 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.340636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:12:46.455240 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.455195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6cw\" (UniqueName: \"kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.455506 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.455268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.556531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.556416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6cw\" (UniqueName: \"kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.556531 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.556511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.557351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.557326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.565786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.565749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6cw\" (UniqueName: \"kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw\") pod \"payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.650102 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.650061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:46.928006 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:46.927976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:12:46.930531 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:12:46.930501 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d1b3d5_3833_435d_94dd_d4aacaa7f6a3.slice/crio-2fffaa6e13b90412a4f0a721847630a452a85b5cbd7c5e4a5cfd43d0590c5c21 WatchSource:0}: Error finding container 2fffaa6e13b90412a4f0a721847630a452a85b5cbd7c5e4a5cfd43d0590c5c21: Status 404 returned error can't find the container with id 2fffaa6e13b90412a4f0a721847630a452a85b5cbd7c5e4a5cfd43d0590c5c21 Sep 16 14:12:47.830754 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:47.830716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" event={"ID":"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3","Type":"ContainerStarted","Data":"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9"} Sep 16 14:12:47.831009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:47.830780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" event={"ID":"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3","Type":"ContainerStarted","Data":"2fffaa6e13b90412a4f0a721847630a452a85b5cbd7c5e4a5cfd43d0590c5c21"} Sep 16 14:12:47.831009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:47.830806 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:12:47.851478 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:47.851426 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" podStartSLOduration=1.851408712 podStartE2EDuration="1.851408712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:47.849326314 +0000 UTC m=+1786.889920289" watchObservedRunningTime="2026-09-16 14:12:47.851408712 +0000 UTC m=+1786.892002667" Sep 16 14:12:51.446072 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:51.446020 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:12:58.838576 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:12:58.838492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:13:08.070639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:08.070591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:08.922329 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:08.922289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:09.822241 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:09.822196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:10.679558 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:10.679519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:11.501539 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:11.501483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:12.186925 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.186862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:12.311147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.311106 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:12.864803 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.864762 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz"] Sep 16 14:13:12.868351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.868331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:12.870775 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.870749 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:13:12.870945 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.870793 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:13:12.878454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.878422 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz"] Sep 16 14:13:12.919728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:12.919675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz\" (UID: \"d1e31a0b-29c3-42da-849d-cf1d2da64444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:13.021259 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.021220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz\" (UID: \"d1e31a0b-29c3-42da-849d-cf1d2da64444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:13.021950 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.021926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz\" (UID: \"d1e31a0b-29c3-42da-849d-cf1d2da64444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:13.180835 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.180735 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:13.399032 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.398993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:13.462246 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.462152 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz"] Sep 16 14:13:13.465840 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:13:13.465811 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e31a0b_29c3_42da_849d_cf1d2da64444.slice/crio-7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983 WatchSource:0}: Error finding container 7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983: Status 404 returned error can't find the container with id 7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983 Sep 16 14:13:13.974916 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.974846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" event={"ID":"d1e31a0b-29c3-42da-849d-cf1d2da64444","Type":"ContainerStarted","Data":"eaaf8e0484f6b01b41a62bd69809185ddb9a1434d03884258a6bc1fd810c6123"} Sep 16 14:13:13.975316 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.974934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" event={"ID":"d1e31a0b-29c3-42da-849d-cf1d2da64444","Type":"ContainerStarted","Data":"7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983"} Sep 16 14:13:13.993004 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:13.992940 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" podStartSLOduration=1.992922761 podStartE2EDuration="1.992922761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:13.991173426 +0000 UTC m=+1813.031767381" watchObservedRunningTime="2026-09-16 14:13:13.992922761 +0000 UTC m=+1813.033516757" Sep 16 14:13:14.259232 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:14.259148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:14.982589 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:14.982547 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1e31a0b-29c3-42da-849d-cf1d2da64444" containerID="eaaf8e0484f6b01b41a62bd69809185ddb9a1434d03884258a6bc1fd810c6123" exitCode=0 Sep 16 14:13:14.983024 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:14.982617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" event={"ID":"d1e31a0b-29c3-42da-849d-cf1d2da64444","Type":"ContainerDied","Data":"eaaf8e0484f6b01b41a62bd69809185ddb9a1434d03884258a6bc1fd810c6123"} Sep 16 14:13:15.106435 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:15.106394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:15.835763 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:15.835725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:16.134330 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.134299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:16.254608 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.254570 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca\") pod \"d1e31a0b-29c3-42da-849d-cf1d2da64444\" (UID: \"d1e31a0b-29c3-42da-849d-cf1d2da64444\") " Sep 16 14:13:16.255005 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.254980 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1e31a0b-29c3-42da-849d-cf1d2da64444" (UID: "d1e31a0b-29c3-42da-849d-cf1d2da64444"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:16.356042 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.355997 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e31a0b-29c3-42da-849d-cf1d2da64444-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:16.892733 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.892683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:16.994780 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.994747 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" Sep 16 14:13:16.994780 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.994777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz" event={"ID":"d1e31a0b-29c3-42da-849d-cf1d2da64444","Type":"ContainerDied","Data":"7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983"} Sep 16 14:13:16.995024 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:16.994829 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2691769b32f31d334cdbf91c8f9fe61884a73136feb098b62b35c36447b983" Sep 16 14:13:17.665675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:17.665637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:24.041338 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.041298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:24.081886 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.081828 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:13:24.082109 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.082083 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" podUID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" containerName="payload-pre-processing" containerID="cri-o://76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9" gracePeriod=30 Sep 16 14:13:24.287746 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.287683 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz"] Sep 16 14:13:24.293702 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.293603 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-29812a-ttdjz"] Sep 16 14:13:24.349397 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.349366 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:13:24.438106 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.438065 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6cw\" (UniqueName: \"kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw\") pod \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " Sep 16 14:13:24.438337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.438120 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume\") pod \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\" (UID: \"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3\") " Sep 16 14:13:24.438559 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.438532 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" (UID: "33d1b3d5-3833-435d-94dd-d4aacaa7f6a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:24.438648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.438629 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:24.440343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.440320 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw" (OuterVolumeSpecName: "kube-api-access-6q6cw") pod "33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" (UID: "33d1b3d5-3833-435d-94dd-d4aacaa7f6a3"). InnerVolumeSpecName "kube-api-access-6q6cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:24.539607 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:24.539568 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6cw\" (UniqueName: \"kubernetes.io/projected/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3-kube-api-access-6q6cw\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:25.038353 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.038319 2559 generic.go:356] "Generic (PLEG): container finished" podID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" containerID="76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9" exitCode=0 Sep 16 14:13:25.038552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.038363 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" Sep 16 14:13:25.038552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.038407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" event={"ID":"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3","Type":"ContainerDied","Data":"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9"} Sep 16 14:13:25.038552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.038455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk" event={"ID":"33d1b3d5-3833-435d-94dd-d4aacaa7f6a3","Type":"ContainerDied","Data":"2fffaa6e13b90412a4f0a721847630a452a85b5cbd7c5e4a5cfd43d0590c5c21"} Sep 16 14:13:25.038552 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.038477 2559 scope.go:117] "RemoveContainer" containerID="76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9" Sep 16 14:13:25.047466 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.047443 2559 scope.go:117] "RemoveContainer" containerID="76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9" Sep 16 14:13:25.047747 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:13:25.047727 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9\": container with ID starting with 76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9 not found: ID does not exist" containerID="76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9" Sep 16 14:13:25.047793 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.047755 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9"} err="failed to get container status \"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9\": rpc error: code = NotFound desc = could not find container \"76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9\": container with ID starting with 76cd234b846bd300d8f7a0a1b1f87909a529ab32bd1a5dff87299ee80bca4ed9 not found: ID does not exist" Sep 16 14:13:25.062411 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.062379 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:13:25.070780 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.070750 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-29812a-5f4459b878-dlkhk"] Sep 16 14:13:25.472550 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.472514 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" path="/var/lib/kubelet/pods/33d1b3d5-3833-435d-94dd-d4aacaa7f6a3/volumes" Sep 16 14:13:25.472924 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:25.472911 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e31a0b-29c3-42da-849d-cf1d2da64444" path="/var/lib/kubelet/pods/d1e31a0b-29c3-42da-849d-cf1d2da64444/volumes" Sep 16 14:13:27.141861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:27.141825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:27.290989 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:27.290951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:28.261918 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:28.261844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:28.741616 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:28.741572 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:13:31.226260 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:31.226223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:32.034071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:32.034033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:32.135138 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:32.135102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:35.096147 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096106 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk"] Sep 16 14:13:35.096555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096534 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" containerName="payload-pre-processing" Sep 16 14:13:35.096555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096546 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" containerName="payload-pre-processing" Sep 16 14:13:35.096555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096556 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e31a0b-29c3-42da-849d-cf1d2da64444" containerName="revoke-keys" Sep 16 14:13:35.096665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096561 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e31a0b-29c3-42da-849d-cf1d2da64444" containerName="revoke-keys" Sep 16 14:13:35.096665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096633 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d1b3d5-3833-435d-94dd-d4aacaa7f6a3" containerName="payload-pre-processing" Sep 16 14:13:35.096665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.096641 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e31a0b-29c3-42da-849d-cf1d2da64444" containerName="revoke-keys" Sep 16 14:13:35.099706 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.099677 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:35.102109 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.102087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:13:35.102289 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.102214 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:13:35.107136 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.107104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk"] Sep 16 14:13:35.251143 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.251091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk\" (UID: \"7b18dcba-1d0b-42e6-bee6-201fd21094ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:35.352431 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.352335 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk\" (UID: \"7b18dcba-1d0b-42e6-bee6-201fd21094ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:35.353023 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.353005 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk\" (UID: \"7b18dcba-1d0b-42e6-bee6-201fd21094ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:35.412269 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.412229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:35.684855 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.684772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk"] Sep 16 14:13:35.688138 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:13:35.688104 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b18dcba_1d0b_42e6_bee6_201fd21094ba.slice/crio-dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944 WatchSource:0}: Error finding container dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944: Status 404 returned error can't find the container with id dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944 Sep 16 14:13:35.978600 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:35.978565 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:36.098821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:36.098776 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" event={"ID":"7b18dcba-1d0b-42e6-bee6-201fd21094ba","Type":"ContainerStarted","Data":"71813d1a342c68d45b2f6796e53428d8d84c82296ca41e2ae84dbd0766612785"} Sep 16 14:13:36.099351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:36.098843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" event={"ID":"7b18dcba-1d0b-42e6-bee6-201fd21094ba","Type":"ContainerStarted","Data":"dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944"} Sep 16 14:13:36.126899 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:36.126804 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" podStartSLOduration=1.126784348 podStartE2EDuration="1.126784348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:13:36.123420699 +0000 UTC m=+1835.164014654" watchObservedRunningTime="2026-09-16 14:13:36.126784348 +0000 UTC m=+1835.167378302" Sep 16 14:13:36.909687 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:36.909652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:37.107527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:37.107487 2559 generic.go:356] "Generic (PLEG): container finished" podID="7b18dcba-1d0b-42e6-bee6-201fd21094ba" containerID="71813d1a342c68d45b2f6796e53428d8d84c82296ca41e2ae84dbd0766612785" exitCode=0 Sep 16 14:13:37.108009 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:37.107561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" event={"ID":"7b18dcba-1d0b-42e6-bee6-201fd21094ba","Type":"ContainerDied","Data":"71813d1a342c68d45b2f6796e53428d8d84c82296ca41e2ae84dbd0766612785"} Sep 16 14:13:37.789359 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:37.789323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:38.250821 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:38.250791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:38.383450 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:38.383354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca\") pod \"7b18dcba-1d0b-42e6-bee6-201fd21094ba\" (UID: \"7b18dcba-1d0b-42e6-bee6-201fd21094ba\") " Sep 16 14:13:38.383824 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:38.383792 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b18dcba-1d0b-42e6-bee6-201fd21094ba" (UID: "7b18dcba-1d0b-42e6-bee6-201fd21094ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:38.484879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:38.484840 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b18dcba-1d0b-42e6-bee6-201fd21094ba-maas-api-service-ca\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:38.669317 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:38.669231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:39.121074 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:39.121037 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" Sep 16 14:13:39.121264 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:39.121072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk" event={"ID":"7b18dcba-1d0b-42e6-bee6-201fd21094ba","Type":"ContainerDied","Data":"dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944"} Sep 16 14:13:39.121264 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:39.121130 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7cd572b773df2bae433b59e5107ab6fb74928a8917da53eb8f7a5546069944" Sep 16 14:13:39.512021 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:39.511984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:47.077687 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:47.077647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:48.246921 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:48.246863 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:13:50.723856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.723822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:50.752595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.752553 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:13:50.752919 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.752847 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" podUID="d1bfb05c-2679-458e-8e68-31c658ab7a91" containerName="payload-pre-processing" containerID="cri-o://370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808" gracePeriod=30 Sep 16 14:13:50.831192 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.831123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:50.926536 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.926495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:50.991639 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.991544 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk"] Sep 16 14:13:50.995241 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:50.995209 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cabdc2-vvkhk"] Sep 16 14:13:51.009170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.009131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:51.025042 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.025014 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:13:51.097601 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.097564 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume\") pod \"d1bfb05c-2679-458e-8e68-31c658ab7a91\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " Sep 16 14:13:51.097794 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.097776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5zz2\" (UniqueName: \"kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2\") pod \"d1bfb05c-2679-458e-8e68-31c658ab7a91\" (UID: \"d1bfb05c-2679-458e-8e68-31c658ab7a91\") " Sep 16 14:13:51.098086 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.098062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1bfb05c-2679-458e-8e68-31c658ab7a91" (UID: "d1bfb05c-2679-458e-8e68-31c658ab7a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:51.098199 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.098184 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bfb05c-2679-458e-8e68-31c658ab7a91-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.099966 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.099944 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2" (OuterVolumeSpecName: "kube-api-access-x5zz2") pod "d1bfb05c-2679-458e-8e68-31c658ab7a91" (UID: "d1bfb05c-2679-458e-8e68-31c658ab7a91"). InnerVolumeSpecName "kube-api-access-x5zz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:51.186932 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.186898 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1bfb05c-2679-458e-8e68-31c658ab7a91" containerID="370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808" exitCode=0 Sep 16 14:13:51.187121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.186944 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" Sep 16 14:13:51.187121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.186958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" event={"ID":"d1bfb05c-2679-458e-8e68-31c658ab7a91","Type":"ContainerDied","Data":"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808"} Sep 16 14:13:51.187121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.187016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj" event={"ID":"d1bfb05c-2679-458e-8e68-31c658ab7a91","Type":"ContainerDied","Data":"ab4e70662c31efc33569eab7bf6fd6e2842b82756a7c1dfe6daa26f16b45f0c1"} Sep 16 14:13:51.187121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.187043 2559 scope.go:117] "RemoveContainer" containerID="370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808" Sep 16 14:13:51.196589 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.196567 2559 scope.go:117] "RemoveContainer" containerID="370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808" Sep 16 14:13:51.196967 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:13:51.196946 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808\": container with ID starting with 370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808 not found: ID does not exist" containerID="370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808" Sep 16 14:13:51.197057 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.196977 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808"} err="failed to get container status \"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808\": rpc error: code = NotFound desc = could not find container \"370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808\": container with ID starting with 370320f2a509612026919a9950a52e8650d3e539541c776cfa16fe5d07ecb808 not found: ID does not exist" Sep 16 14:13:51.199247 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.199227 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zz2\" (UniqueName: \"kubernetes.io/projected/d1bfb05c-2679-458e-8e68-31c658ab7a91-kube-api-access-x5zz2\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.210094 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.210059 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:13:51.216915 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.216886 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cabdc2-59c8b768c4-cnmkj"] Sep 16 14:13:51.359142 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.359059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:51.428525 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.428490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:51.473342 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.473305 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b18dcba-1d0b-42e6-bee6-201fd21094ba" path="/var/lib/kubelet/pods/7b18dcba-1d0b-42e6-bee6-201fd21094ba/volumes" Sep 16 14:13:51.473682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:51.473669 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bfb05c-2679-458e-8e68-31c658ab7a91" path="/var/lib/kubelet/pods/d1bfb05c-2679-458e-8e68-31c658ab7a91/volumes" Sep 16 14:13:52.333546 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:52.333513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:53.225305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:53.225267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:54.153938 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:54.153845 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:13:56.332846 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:56.332804 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:13:56.428665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:13:56.428621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:01.257775 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:01.257739 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:14:11.353319 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.353262 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" podUID="7f2efa78-b160-4f76-856d-9314fb222f8b" containerName="main" containerID="cri-o://1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c" gracePeriod=45 Sep 16 14:14:11.610992 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.610924 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:14:11.713181 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713181 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713230 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713428 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713527 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713455 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:11.713635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713524 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:11.713635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713533 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home" (OuterVolumeSpecName: "home") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:11.713635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713529 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tddvh\" (UniqueName: \"kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh\") pod \"7f2efa78-b160-4f76-856d-9314fb222f8b\" (UID: \"7f2efa78-b160-4f76-856d-9314fb222f8b\") " Sep 16 14:14:11.713856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713835 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:11.713856 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713853 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:11.714000 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.713861 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:11.715632 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.715609 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:11.715728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.715660 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh" (OuterVolumeSpecName: "kube-api-access-tddvh") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "kube-api-access-tddvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:11.715728 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.715674 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm" (OuterVolumeSpecName: "dshm") pod "7f2efa78-b160-4f76-856d-9314fb222f8b" (UID: "7f2efa78-b160-4f76-856d-9314fb222f8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:11.814549 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.814511 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2efa78-b160-4f76-856d-9314fb222f8b-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:11.814549 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.814542 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2efa78-b160-4f76-856d-9314fb222f8b-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:11.814549 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:11.814551 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddvh\" (UniqueName: \"kubernetes.io/projected/7f2efa78-b160-4f76-856d-9314fb222f8b-kube-api-access-tddvh\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:12.306329 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.306294 2559 generic.go:356] "Generic (PLEG): container finished" podID="7f2efa78-b160-4f76-856d-9314fb222f8b" containerID="1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c" exitCode=0 Sep 16 14:14:12.306562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.306341 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" Sep 16 14:14:12.306562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.306375 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" event={"ID":"7f2efa78-b160-4f76-856d-9314fb222f8b","Type":"ContainerDied","Data":"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c"} Sep 16 14:14:12.306562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.306426 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km" event={"ID":"7f2efa78-b160-4f76-856d-9314fb222f8b","Type":"ContainerDied","Data":"4a2a9b72d1729ffcc8193ec5761f7917aa1a9180e9aefd1e318a0a678a32daad"} Sep 16 14:14:12.306562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.306449 2559 scope.go:117] "RemoveContainer" containerID="1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c" Sep 16 14:14:12.316302 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.316283 2559 scope.go:117] "RemoveContainer" containerID="1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c" Sep 16 14:14:12.316608 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:14:12.316590 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c\": container with ID starting with 1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c not found: ID does not exist" containerID="1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c" Sep 16 14:14:12.316657 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.316618 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c"} err="failed to get container status \"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c\": rpc error: code = NotFound desc = could not find container \"1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c\": container with ID starting with 1596990f60718285e0dce915efb5ed37d090dfd67e3c89227c2c2a377d3f3b3c not found: ID does not exist" Sep 16 14:14:12.330455 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.330421 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:14:12.335601 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:12.335573 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cabdc2/ipp-route-cabdc2-kserve-7f4fbd5655-kf6km"] Sep 16 14:14:13.471745 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:13.471708 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2efa78-b160-4f76-856d-9314fb222f8b" path="/var/lib/kubelet/pods/7f2efa78-b160-4f76-856d-9314fb222f8b/volumes" Sep 16 14:14:20.675668 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:20.675627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:20.775901 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:20.775855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:22.184029 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:22.183993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:36.057216 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:36.057121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:36.149645 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:36.149606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:39.906049 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:39.906012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:46.157933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:46.157895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:46.230347 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:46.230311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:46.432596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:46.432510 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:14:46.432752 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:46.432714 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" containerID="cri-o://03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3" gracePeriod=30 Sep 16 14:14:46.456440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:46.456386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:47.116299 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:47.116263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:48.183852 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:48.183808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:49.140357 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:49.140315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:50.055584 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:50.055549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:50.842226 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:50.842183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:51.699637 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.699609 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:14:51.896080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896044 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896168 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896277 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896195 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896307 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ngv9\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896337 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896389 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert\") pod \"775682c6-f582-455c-bbf2-b5c2244b7cb4\" (UID: \"775682c6-f582-455c-bbf2-b5c2244b7cb4\") " Sep 16 14:14:51.896507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896412 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:51.896747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896535 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:51.896747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:51.896747 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896623 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:51.896940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896780 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.896940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896799 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.896940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896813 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.896940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896826 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.896940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.896831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:51.898524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.898488 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9" (OuterVolumeSpecName: "kube-api-access-7ngv9") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "kube-api-access-7ngv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:51.898654 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.898629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:14:51.898955 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.898935 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:14:51.898955 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.898947 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "775682c6-f582-455c-bbf2-b5c2244b7cb4" (UID: "775682c6-f582-455c-bbf2-b5c2244b7cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:51.998101 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.998003 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ngv9\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-kube-api-access-7ngv9\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.998101 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.998039 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.998101 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.998049 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775682c6-f582-455c-bbf2-b5c2244b7cb4-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.998101 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.998057 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:51.998101 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:51.998064 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775682c6-f582-455c-bbf2-b5c2244b7cb4-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:52.524468 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.524432 2559 generic.go:356] "Generic (PLEG): container finished" podID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerID="03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3" exitCode=0 Sep 16 14:14:52.524663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.524476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" Sep 16 14:14:52.524663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.524515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerDied","Data":"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3"} Sep 16 14:14:52.524663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.524567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw" event={"ID":"775682c6-f582-455c-bbf2-b5c2244b7cb4","Type":"ContainerDied","Data":"724a8b750cd1f4f3a7a8f247265f0cdcf9d66121dc52fcdb643de296b5129308"} Sep 16 14:14:52.524663 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.524590 2559 scope.go:117] "RemoveContainer" containerID="03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3" Sep 16 14:14:52.534348 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.534329 2559 scope.go:117] "RemoveContainer" containerID="e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97" Sep 16 14:14:52.542442 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.542424 2559 scope.go:117] "RemoveContainer" containerID="03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3" Sep 16 14:14:52.542734 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:14:52.542716 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3\": container with ID starting with 03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3 not found: ID does not exist" containerID="03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3" Sep 16 14:14:52.542807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.542747 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3"} err="failed to get container status \"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3\": rpc error: code = NotFound desc = could not find container \"03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3\": container with ID starting with 03f1a44825c42d2790b0b8e999cf3eb9f14b20816a53d6f1206c99deef3a55f3 not found: ID does not exist" Sep 16 14:14:52.542807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.542774 2559 scope.go:117] "RemoveContainer" containerID="e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97" Sep 16 14:14:52.543059 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:14:52.543040 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97\": container with ID starting with e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97 not found: ID does not exist" containerID="e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97" Sep 16 14:14:52.543105 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.543064 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97"} err="failed to get container status \"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97\": rpc error: code = NotFound desc = could not find container \"e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97\": container with ID starting with e4c2f69197008c7b8531ecd5ff62a90124492c2f27b0af9c31f47dae3a2f6c97 not found: ID does not exist" Sep 16 14:14:52.548722 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.548699 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:14:52.553784 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:52.553761 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cabdc2-openshift-default-54fd4c75fb-kx9mw"] Sep 16 14:14:53.478640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:53.478597 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" path="/var/lib/kubelet/pods/775682c6-f582-455c-bbf2-b5c2244b7cb4/volumes" Sep 16 14:14:57.566450 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.566409 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:14:57.567655 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.567618 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" podUID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" containerName="payload-pre-processing" containerID="cri-o://41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059" gracePeriod=30 Sep 16 14:14:57.627205 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:14:57.627170 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d055879_6bb3_47a8_9c6f_eb61b9f113a0.slice/crio-conmon-41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059.scope\": RecentStats: unable to find data in memory cache]" Sep 16 14:14:57.826727 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.826662 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:14:57.850170 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.850132 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgcmh\" (UniqueName: \"kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh\") pod \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " Sep 16 14:14:57.850358 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.850215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume\") pod \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\" (UID: \"2d055879-6bb3-47a8-9c6f-eb61b9f113a0\") " Sep 16 14:14:57.850638 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.850588 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d055879-6bb3-47a8-9c6f-eb61b9f113a0" (UID: "2d055879-6bb3-47a8-9c6f-eb61b9f113a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:57.852440 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.852406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh" (OuterVolumeSpecName: "kube-api-access-fgcmh") pod "2d055879-6bb3-47a8-9c6f-eb61b9f113a0" (UID: "2d055879-6bb3-47a8-9c6f-eb61b9f113a0"). InnerVolumeSpecName "kube-api-access-fgcmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:57.951972 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.951926 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcmh\" (UniqueName: \"kubernetes.io/projected/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-kube-api-access-fgcmh\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:57.951972 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:57.951963 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d055879-6bb3-47a8-9c6f-eb61b9f113a0-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:14:58.559607 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.559570 2559 generic.go:356] "Generic (PLEG): container finished" podID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" containerID="41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059" exitCode=0 Sep 16 14:14:58.559807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.559612 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" Sep 16 14:14:58.559807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.559652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" event={"ID":"2d055879-6bb3-47a8-9c6f-eb61b9f113a0","Type":"ContainerDied","Data":"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059"} Sep 16 14:14:58.559807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.559711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m" event={"ID":"2d055879-6bb3-47a8-9c6f-eb61b9f113a0","Type":"ContainerDied","Data":"0fb2499fe0fdc4c2d4ec8af89f271a469175f65d0686e022ceeae8199ca91c3a"} Sep 16 14:14:58.559807 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.559735 2559 scope.go:117] "RemoveContainer" containerID="41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059" Sep 16 14:14:58.569443 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.569253 2559 scope.go:117] "RemoveContainer" containerID="41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059" Sep 16 14:14:58.569670 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:14:58.569557 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059\": container with ID starting with 41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059 not found: ID does not exist" containerID="41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059" Sep 16 14:14:58.569670 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.569583 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059"} err="failed to get container status \"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059\": rpc error: code = NotFound desc = could not find container \"41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059\": container with ID starting with 41b350ef75e35663d081db246c89c5a4e6ab3cd0e7ab16f8e350a1ea57836059 not found: ID does not exist" Sep 16 14:14:58.591553 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.591512 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:14:58.596939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:58.596906 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c8daa-6c699fc6fd-wpc6m"] Sep 16 14:14:59.471682 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:59.471645 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" path="/var/lib/kubelet/pods/2d055879-6bb3-47a8-9c6f-eb61b9f113a0/volumes" Sep 16 14:14:59.503426 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:59.503389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:14:59.613454 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:14:59.613416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:00.149766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.149726 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r"] Sep 16 14:15:00.150278 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150258 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" containerName="payload-pre-processing" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150280 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" containerName="payload-pre-processing" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150289 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b18dcba-1d0b-42e6-bee6-201fd21094ba" containerName="revoke-keys" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150296 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b18dcba-1d0b-42e6-bee6-201fd21094ba" containerName="revoke-keys" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150310 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bfb05c-2679-458e-8e68-31c658ab7a91" containerName="payload-pre-processing" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150316 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bfb05c-2679-458e-8e68-31c658ab7a91" containerName="payload-pre-processing" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150331 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2efa78-b160-4f76-856d-9314fb222f8b" containerName="main" Sep 16 14:15:00.150343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150336 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2efa78-b160-4f76-856d-9314fb222f8b" containerName="main" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150354 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150361 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150366 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150452 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150460 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="775682c6-f582-455c-bbf2-b5c2244b7cb4" containerName="istio-proxy" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150468 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bfb05c-2679-458e-8e68-31c658ab7a91" containerName="payload-pre-processing" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150473 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2efa78-b160-4f76-856d-9314fb222f8b" containerName="main" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150479 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b18dcba-1d0b-42e6-bee6-201fd21094ba" containerName="revoke-keys" Sep 16 14:15:00.150551 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.150484 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d055879-6bb3-47a8-9c6f-eb61b9f113a0" containerName="payload-pre-processing" Sep 16 14:15:00.163612 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.163579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r"] Sep 16 14:15:00.163798 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.163712 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" Sep 16 14:15:00.175173 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.175147 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\"" Sep 16 14:15:00.176184 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.176162 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" Sep 16 14:15:00.468347 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.468321 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r"] Sep 16 14:15:00.470714 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:15:00.470681 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d18601_cbdc_4a8b_a68f_d23b0bb765a8.slice/crio-2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82 WatchSource:0}: Error finding container 2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82: Status 404 returned error can't find the container with id 2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82 Sep 16 14:15:00.496509 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.496474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:00.572508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.572473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" event={"ID":"33d18601-cbdc-4a8b-a68f-d23b0bb765a8","Type":"ContainerStarted","Data":"2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82"} Sep 16 14:15:00.857199 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:00.857159 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:15:01.579640 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:01.579605 2559 generic.go:356] "Generic (PLEG): container finished" podID="33d18601-cbdc-4a8b-a68f-d23b0bb765a8" containerID="768db9362240a6cda319ff0f7840e03e54cee919b14fb79297fce74b634af108" exitCode=0 Sep 16 14:15:01.579843 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:01.579693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" event={"ID":"33d18601-cbdc-4a8b-a68f-d23b0bb765a8","Type":"ContainerDied","Data":"768db9362240a6cda319ff0f7840e03e54cee919b14fb79297fce74b634af108"} Sep 16 14:15:01.945861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:01.945825 2559 scope.go:117] "RemoveContainer" containerID="d03496c0469c012c9481efd8081ee211013a0ab7017d985a9c210ef411d41728" Sep 16 14:15:01.955284 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:01.955236 2559 scope.go:117] "RemoveContainer" containerID="eb7509030383561a418cbd4c184299829bfdf42009e5db2527128701475da255" Sep 16 14:15:02.720587 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:02.720555 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" Sep 16 14:15:03.591555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:03.591517 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" event={"ID":"33d18601-cbdc-4a8b-a68f-d23b0bb765a8","Type":"ContainerDied","Data":"2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82"} Sep 16 14:15:03.591555 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:03.591551 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r" Sep 16 14:15:03.592028 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:03.591571 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc24657a0ff2e2156572ad79431b056c00a31c5473313b5d1b0539ac1235a82" Sep 16 14:15:04.745967 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:04.745934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:04.997085 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:04.996976 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:15:04.997292 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:04.997264 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" containerID="cri-o://61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49" gracePeriod=30 Sep 16 14:15:05.014416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:05.014379 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:05.115400 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:05.115359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:05.540546 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:05.540512 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:06.660784 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:06.660738 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:07.446015 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:07.445980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:08.380241 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:08.380169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:10.258246 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.258165 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:15:10.373524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373473 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373524 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373531 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373556 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373581 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373597 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373627 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373661 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373696 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.373795 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373725 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d6tr\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr\") pod \"3c985712-a57e-4ee4-8269-604d564c1e5b\" (UID: \"3c985712-a57e-4ee4-8269-604d564c1e5b\") " Sep 16 14:15:10.374174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373951 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:10.374174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.373991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:10.374174 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374059 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:10.374393 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374211 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.374393 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374221 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:10.374393 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374232 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.374393 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374248 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.374607 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.374389 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:15:10.376038 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.376012 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:15:10.376038 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.376022 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr" (OuterVolumeSpecName: "kube-api-access-4d6tr") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "kube-api-access-4d6tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:10.376294 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.376271 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:10.376294 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.376276 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c985712-a57e-4ee4-8269-604d564c1e5b" (UID: "3c985712-a57e-4ee4-8269-604d564c1e5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:10.475197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475149 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.475197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475187 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.475197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475196 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c985712-a57e-4ee4-8269-604d564c1e5b-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.475197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475204 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.475197 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475212 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c985712-a57e-4ee4-8269-604d564c1e5b-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.475508 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.475220 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6tr\" (UniqueName: \"kubernetes.io/projected/3c985712-a57e-4ee4-8269-604d564c1e5b-kube-api-access-4d6tr\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:10.629819 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.629726 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerID="61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49" exitCode=0 Sep 16 14:15:10.629819 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.629778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" Sep 16 14:15:10.629819 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.629786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerDied","Data":"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49"} Sep 16 14:15:10.630087 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.629844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl" event={"ID":"3c985712-a57e-4ee4-8269-604d564c1e5b","Type":"ContainerDied","Data":"bd7ddc22404bdec4fcb0adbbd02b4b6dccb80702e89e5e42674073a80b35a66b"} Sep 16 14:15:10.630087 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.629899 2559 scope.go:117] "RemoveContainer" containerID="61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49" Sep 16 14:15:10.643683 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.643658 2559 scope.go:117] "RemoveContainer" containerID="8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66" Sep 16 14:15:10.652564 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.652539 2559 scope.go:117] "RemoveContainer" containerID="61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49" Sep 16 14:15:10.652908 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:15:10.652863 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49\": container with ID starting with 61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49 not found: ID does not exist" containerID="61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49" Sep 16 14:15:10.652977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.652918 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49"} err="failed to get container status \"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49\": rpc error: code = NotFound desc = could not find container \"61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49\": container with ID starting with 61b481a1a17c6add2123681c51a3678c7ef4cb1b36fcbbb15f90c87afe0acc49 not found: ID does not exist" Sep 16 14:15:10.652977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.652938 2559 scope.go:117] "RemoveContainer" containerID="8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66" Sep 16 14:15:10.653252 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:15:10.653233 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66\": container with ID starting with 8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66 not found: ID does not exist" containerID="8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66" Sep 16 14:15:10.653311 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.653254 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66"} err="failed to get container status \"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66\": rpc error: code = NotFound desc = could not find container \"8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66\": container with ID starting with 8f98fe9681b7616c6421c9876380eff341ab941e9385639fdb18f7be7cfadc66 not found: ID does not exist" Sep 16 14:15:10.658604 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.658555 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:15:10.661037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:10.661009 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9c8daa-openshift-default-9fb6c75b4-24wxl"] Sep 16 14:15:11.472786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:11.472748 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" path="/var/lib/kubelet/pods/3c985712-a57e-4ee4-8269-604d564c1e5b/volumes" Sep 16 14:15:15.968340 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:15.968305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:15.979842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:15.979806 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r"] Sep 16 14:15:15.982793 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:15.982759 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-46224d-29826135-2f64r"] Sep 16 14:15:16.116785 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:16.116745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:16.205755 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:16.205717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:16.286955 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:16.286824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:16.665127 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:16.665024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:16.766123 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:16.766072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:17.471808 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:17.471770 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d18601-cbdc-4a8b-a68f-d23b0bb765a8" path="/var/lib/kubelet/pods/33d18601-cbdc-4a8b-a68f-d23b0bb765a8/volumes" Sep 16 14:15:19.050325 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:19.050287 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:15:19.659923 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:19.659860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:23.531190 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:23.531150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:23.600558 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:23.600524 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:24.598287 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:24.598249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:24.701423 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:24.701386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:25.628999 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:25.628964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:26.519336 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:26.519297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:34.801387 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:34.801326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:34.842787 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:34.842750 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:15:34.843104 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:34.843072 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" podUID="102d71c7-0324-493d-bdc6-fc62c565c75a" containerName="payload-pre-processing" containerID="cri-o://f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439" gracePeriod=30 Sep 16 14:15:34.900707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:34.900651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:34.999985 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:34.999942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:35.101329 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.101298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:35.106349 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.106318 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:15:35.201427 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.201381 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vhxc\" (UniqueName: \"kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc\") pod \"102d71c7-0324-493d-bdc6-fc62c565c75a\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " Sep 16 14:15:35.201648 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.201517 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume\") pod \"102d71c7-0324-493d-bdc6-fc62c565c75a\" (UID: \"102d71c7-0324-493d-bdc6-fc62c565c75a\") " Sep 16 14:15:35.201953 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.201917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "102d71c7-0324-493d-bdc6-fc62c565c75a" (UID: "102d71c7-0324-493d-bdc6-fc62c565c75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:15:35.203781 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.203750 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc" (OuterVolumeSpecName: "kube-api-access-9vhxc") pod "102d71c7-0324-493d-bdc6-fc62c565c75a" (UID: "102d71c7-0324-493d-bdc6-fc62c565c75a"). InnerVolumeSpecName "kube-api-access-9vhxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:35.302631 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.302587 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102d71c7-0324-493d-bdc6-fc62c565c75a-plugins-config-volume\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:35.302631 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.302621 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vhxc\" (UniqueName: \"kubernetes.io/projected/102d71c7-0324-493d-bdc6-fc62c565c75a-kube-api-access-9vhxc\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:35.420058 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.419962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:35.526410 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.526373 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:35.758944 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.758909 2559 generic.go:356] "Generic (PLEG): container finished" podID="102d71c7-0324-493d-bdc6-fc62c565c75a" containerID="f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439" exitCode=0 Sep 16 14:15:35.759140 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.758962 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" Sep 16 14:15:35.759140 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.759001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" event={"ID":"102d71c7-0324-493d-bdc6-fc62c565c75a","Type":"ContainerDied","Data":"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439"} Sep 16 14:15:35.759140 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.759059 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27" event={"ID":"102d71c7-0324-493d-bdc6-fc62c565c75a","Type":"ContainerDied","Data":"0f1b336940254743015846bdad79c59eb08f853896fb7b6bfd22cc0c3cc2eea3"} Sep 16 14:15:35.759140 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.759089 2559 scope.go:117] "RemoveContainer" containerID="f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439" Sep 16 14:15:35.768037 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.768014 2559 scope.go:117] "RemoveContainer" containerID="f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439" Sep 16 14:15:35.768373 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:15:35.768347 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439\": container with ID starting with f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439 not found: ID does not exist" containerID="f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439" Sep 16 14:15:35.768421 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.768382 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439"} err="failed to get container status \"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439\": rpc error: code = NotFound desc = could not find container \"f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439\": container with ID starting with f7488bc7e1cada2dfb378f34e247cd396f3f52094935bc4908061554abc68439 not found: ID does not exist" Sep 16 14:15:35.787612 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.787571 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:15:35.793824 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:35.793795 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b4884e-86799bc9fd-ccs27"] Sep 16 14:15:37.472404 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:37.472366 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102d71c7-0324-493d-bdc6-fc62c565c75a" path="/var/lib/kubelet/pods/102d71c7-0324-493d-bdc6-fc62c565c75a/volumes" Sep 16 14:15:37.738175 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:37.738085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:15:41.863884 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:41.863842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:41.969571 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:41.969535 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:42.112117 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:42.112075 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:15:42.112365 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:42.112341 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" containerID="cri-o://01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62" gracePeriod=30 Sep 16 14:15:42.145201 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:42.145111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:15:46.439596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:46.439553 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:15:47.359733 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.359697 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:15:47.422966 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.422844 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxngq\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.422966 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.422944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.422975 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.422998 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423029 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423097 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423120 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423218 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423190 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo\") pod \"83422dbd-ebe3-430c-9500-18b9af62f9dc\" (UID: \"83422dbd-ebe3-430c-9500-18b9af62f9dc\") " Sep 16 14:15:47.423556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423320 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:47.423556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423328 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:47.423556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423424 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:15:47.423556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423532 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:47.423777 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423733 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.423777 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423753 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83422dbd-ebe3-430c-9500-18b9af62f9dc-istiod-ca-cert\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.423777 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423766 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-data\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.423919 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423779 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-workload-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.423919 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.423844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:47.425195 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.425172 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq" (OuterVolumeSpecName: "kube-api-access-zxngq") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "kube-api-access-zxngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:47.425301 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.425257 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:15:47.425661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.425637 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:47.425661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.425644 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83422dbd-ebe3-430c-9500-18b9af62f9dc" (UID: "83422dbd-ebe3-430c-9500-18b9af62f9dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:15:47.525305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.525263 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxngq\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-kube-api-access-zxngq\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.525305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.525304 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-credential-socket\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.525305 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.525317 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-envoy\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.525786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.525328 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-token\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.525786 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.525338 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83422dbd-ebe3-430c-9500-18b9af62f9dc-istio-podinfo\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:15:47.824730 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.824689 2559 generic.go:356] "Generic (PLEG): container finished" podID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerID="01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62" exitCode=0 Sep 16 14:15:47.824939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.824756 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" Sep 16 14:15:47.824939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.824772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerDied","Data":"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62"} Sep 16 14:15:47.824939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.824881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7" event={"ID":"83422dbd-ebe3-430c-9500-18b9af62f9dc","Type":"ContainerDied","Data":"80a608044fae192bb5f6bd60827b8c7000fc7986a30c27122a273781ac5265ad"} Sep 16 14:15:47.824939 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.824899 2559 scope.go:117] "RemoveContainer" containerID="01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62" Sep 16 14:15:47.834343 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.834317 2559 scope.go:117] "RemoveContainer" containerID="f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4" Sep 16 14:15:47.843409 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.843364 2559 scope.go:117] "RemoveContainer" containerID="01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62" Sep 16 14:15:47.843933 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:15:47.843903 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62\": container with ID starting with 01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62 not found: ID does not exist" containerID="01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62" Sep 16 14:15:47.844107 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.843946 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62"} err="failed to get container status \"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62\": rpc error: code = NotFound desc = could not find container \"01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62\": container with ID starting with 01778cf0500dcc824f40a051c233ac056dd01d3ada77de29a814e9540d05bb62 not found: ID does not exist" Sep 16 14:15:47.844107 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.843974 2559 scope.go:117] "RemoveContainer" containerID="f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4" Sep 16 14:15:47.844352 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:15:47.844326 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4\": container with ID starting with f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4 not found: ID does not exist" containerID="f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4" Sep 16 14:15:47.844404 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.844365 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4"} err="failed to get container status \"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4\": rpc error: code = NotFound desc = could not find container \"f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4\": container with ID starting with f8c2230377e501048e8bb495d49fff6570858d73c90b624ed03b309e22d107c4 not found: ID does not exist" Sep 16 14:15:47.845533 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.845514 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:15:47.849111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:47.849086 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b4884e-openshift-default-57b8f4ccdd-x7xm7"] Sep 16 14:15:49.470934 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:49.470899 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" path="/var/lib/kubelet/pods/83422dbd-ebe3-430c-9500-18b9af62f9dc/volumes" Sep 16 14:15:57.454124 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:15:57.454040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:01.992681 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:16:01.992647 2559 scope.go:117] "RemoveContainer" containerID="df3079622560551a5c4f0694b9c784db56f412d1b90d3509948b6655caa8f2ad" Sep 16 14:16:07.685503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:16:07.685468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:12.163340 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:16:12.163302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:16:12.244288 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:16:12.244250 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:07.555211 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:07.555171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:08.239083 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:08.239047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:17:09.544599 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:09.544562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:25.011273 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:25.011186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:25.112960 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:25.112922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:32.338707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:32.338666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:17:40.662216 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:40.662177 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:17:46.566269 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:46.566235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:46.660351 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:46.660312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:17:49.670213 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:49.670175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:17:57.065242 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:17:57.065204 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:18:08.943016 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:18:08.942979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:18:17.049842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:18:17.049801 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:18:21.855231 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:18:21.855194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:18:21.921599 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:18:21.921551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:19:49.256623 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:19:49.256583 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:19:57.472262 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:19:57.472228 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:20:02.052933 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:02.052893 2559 scope.go:117] "RemoveContainer" containerID="eaaf8e0484f6b01b41a62bd69809185ddb9a1434d03884258a6bc1fd810c6123" Sep 16 14:20:02.063879 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:02.063813 2559 scope.go:117] "RemoveContainer" containerID="71813d1a342c68d45b2f6796e53428d8d84c82296ca41e2ae84dbd0766612785" Sep 16 14:20:07.439671 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.439633 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:20:07.441028 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.440999 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:20:07.441184 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441173 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:20:07.441272 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441261 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102d71c7-0324-493d-bdc6-fc62c565c75a" containerName="payload-pre-processing" Sep 16 14:20:07.441348 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441339 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="102d71c7-0324-493d-bdc6-fc62c565c75a" containerName="payload-pre-processing" Sep 16 14:20:07.441423 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441414 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.441491 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441483 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.441568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441559 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.441632 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441624 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.441704 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441696 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:20:07.441770 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441762 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:20:07.441854 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441845 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d18601-cbdc-4a8b-a68f-d23b0bb765a8" containerName="cleanup" Sep 16 14:20:07.441953 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.441944 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d18601-cbdc-4a8b-a68f-d23b0bb765a8" containerName="cleanup" Sep 16 14:20:07.442190 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.442176 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:20:07.442278 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.442268 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="102d71c7-0324-493d-bdc6-fc62c565c75a" containerName="payload-pre-processing" Sep 16 14:20:07.442503 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.442402 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.442642 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.442626 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d18601-cbdc-4a8b-a68f-d23b0bb765a8" containerName="cleanup" Sep 16 14:20:07.442718 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.442645 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c985712-a57e-4ee4-8269-604d564c1e5b" containerName="istio-proxy" Sep 16 14:20:07.445061 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.445043 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.447710 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.447683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6fdcecf1-kserve-self-signed-certs\"" Sep 16 14:20:07.450436 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.450412 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:20:07.483155 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.483332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.483332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.483332 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpk9s\" (UniqueName: \"kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.483447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.483447 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.483387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.510912 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.510876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:20:07.583977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.583933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584119 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.583993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584119 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584119 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584119 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpk9s\" (UniqueName: \"kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584333 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584393 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584495 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.584556 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.584501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.586417 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.586392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.586635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.586617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.593946 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.593923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpk9s\" (UniqueName: \"kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s\") pod \"e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:07.755501 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:07.755455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:08.034614 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:08.034580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:20:08.038164 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:20:08.038127 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd48fae1_17eb_476d_9ea5_098334261c4c.slice/crio-f00e052fbc31247ff7aac2b0bb160655a7cc99e4fd6139f6bc44f25dcda7d284 WatchSource:0}: Error finding container f00e052fbc31247ff7aac2b0bb160655a7cc99e4fd6139f6bc44f25dcda7d284: Status 404 returned error can't find the container with id f00e052fbc31247ff7aac2b0bb160655a7cc99e4fd6139f6bc44f25dcda7d284 Sep 16 14:20:08.065078 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:08.065043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" event={"ID":"dd48fae1-17eb-476d-9ea5-098334261c4c","Type":"ContainerStarted","Data":"f00e052fbc31247ff7aac2b0bb160655a7cc99e4fd6139f6bc44f25dcda7d284"} Sep 16 14:20:09.072910 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:09.072848 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" event={"ID":"dd48fae1-17eb-476d-9ea5-098334261c4c","Type":"ContainerStarted","Data":"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194"} Sep 16 14:20:09.107765 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:09.107685 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" podStartSLOduration=2.107663817 podStartE2EDuration="2.107663817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:09.105638306 +0000 UTC m=+2228.146232260" watchObservedRunningTime="2026-09-16 14:20:09.107663817 +0000 UTC m=+2228.148257774" Sep 16 14:20:09.977267 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:09.977225 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:20:17.756457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:17.756416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:17.756457 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:17.756466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:17.769079 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:17.769047 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:18.127861 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:18.127767 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:20:28.619109 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:28.618998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:20:31.712894 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:31.712816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:20:34.575056 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:20:34.575010 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:02.088171 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:02.088131 2559 scope.go:117] "RemoveContainer" containerID="768db9362240a6cda319ff0f7840e03e54cee919b14fb79297fce74b634af108" Sep 16 14:21:15.595350 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:15.595313 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:21:15.625609 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:15.625574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:16.923530 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:16.923496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:21:18.969734 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:18.969694 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:28.367163 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:28.367117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:30.614661 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.614620 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" podUID="dd48fae1-17eb-476d-9ea5-098334261c4c" containerName="main" containerID="cri-o://914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194" gracePeriod=45 Sep 16 14:21:30.864370 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.864340 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:21:30.944202 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944211 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944255 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpk9s\" (UniqueName: \"kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944287 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944412 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944347 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home\") pod \"dd48fae1-17eb-476d-9ea5-098334261c4c\" (UID: \"dd48fae1-17eb-476d-9ea5-098334261c4c\") " Sep 16 14:21:30.944707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944615 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:21:30.944707 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:21:30.944830 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.944757 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home" (OuterVolumeSpecName: "home") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:21:30.946611 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.946581 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:30.946864 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.946835 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm" (OuterVolumeSpecName: "dshm") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:21:30.946996 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:30.946891 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s" (OuterVolumeSpecName: "kube-api-access-mpk9s") pod "dd48fae1-17eb-476d-9ea5-098334261c4c" (UID: "dd48fae1-17eb-476d-9ea5-098334261c4c"). InnerVolumeSpecName "kube-api-access-mpk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:21:31.045223 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045180 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd48fae1-17eb-476d-9ea5-098334261c4c-tls-certs\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.045223 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045213 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-dshm\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.045223 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045223 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpk9s\" (UniqueName: \"kubernetes.io/projected/dd48fae1-17eb-476d-9ea5-098334261c4c-kube-api-access-mpk9s\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.045223 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045232 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-tmp-dir\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.045507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045241 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-model-cache\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.045507 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.045249 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd48fae1-17eb-476d-9ea5-098334261c4c-home\") on node \"ip-10-0-139-34.ec2.internal\" DevicePath \"\"" Sep 16 14:21:31.473766 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.473635 2559 generic.go:356] "Generic (PLEG): container finished" podID="dd48fae1-17eb-476d-9ea5-098334261c4c" containerID="914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194" exitCode=0 Sep 16 14:21:31.474001 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.473780 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" Sep 16 14:21:31.476977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.476934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" event={"ID":"dd48fae1-17eb-476d-9ea5-098334261c4c","Type":"ContainerDied","Data":"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194"} Sep 16 14:21:31.476977 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.476977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2" event={"ID":"dd48fae1-17eb-476d-9ea5-098334261c4c","Type":"ContainerDied","Data":"f00e052fbc31247ff7aac2b0bb160655a7cc99e4fd6139f6bc44f25dcda7d284"} Sep 16 14:21:31.477177 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.477012 2559 scope.go:117] "RemoveContainer" containerID="914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194" Sep 16 14:21:31.487187 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.487168 2559 scope.go:117] "RemoveContainer" containerID="914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194" Sep 16 14:21:31.487487 ip-10-0-139-34 kubenswrapper[2559]: E0916 14:21:31.487464 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194\": container with ID starting with 914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194 not found: ID does not exist" containerID="914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194" Sep 16 14:21:31.487568 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.487494 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194"} err="failed to get container status \"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194\": rpc error: code = NotFound desc = could not find container \"914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194\": container with ID starting with 914bb4ab8270953d23840b676616e4f221dc1ff4d4158cb9f831dcf5d0062194 not found: ID does not exist" Sep 16 14:21:31.505796 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.505754 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:21:31.509887 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:31.509846 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6fdcecf1-kserve-6799b5f64c-q4vq2"] Sep 16 14:21:33.471635 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:33.471596 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd48fae1-17eb-476d-9ea5-098334261c4c" path="/var/lib/kubelet/pods/dd48fae1-17eb-476d-9ea5-098334261c4c/volumes" Sep 16 14:21:35.384675 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:35.384633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:43.184749 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:43.184698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:50.547998 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:50.547954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:21:58.698439 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:21:58.698355 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:22:03.460416 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:03.460377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8msjx"] Sep 16 14:22:06.537076 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:06.537032 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54f85fdc87-ccwqh_44ea1263-b681-4bfe-ae12-73e036367a72/authorino/0.log" Sep 16 14:22:12.670916 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:12.670885 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8h9d4_93d4bf31-20f4-4f5d-ac8e-f4378f782545/manager/0.log" Sep 16 14:22:12.783071 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:12.783035 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-7sfbx_4a92a250-a4cc-4c2f-98ca-c5aa03acc32b/copy-manifests/0.log" Sep 16 14:22:12.797542 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:12.797506 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-7sfbx_4a92a250-a4cc-4c2f-98ca-c5aa03acc32b/manager/0.log" Sep 16 14:22:14.592211 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:14.592174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54f85fdc87-ccwqh_44ea1263-b681-4bfe-ae12-73e036367a72/authorino/0.log" Sep 16 14:22:14.954410 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:14.954379 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n44pr_e3d44aa3-6f60-45ce-ac22-80726fd6aa59/kuadrant-console-plugin/0.log" Sep 16 14:22:15.202120 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:15.202092 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8msjx_007c82fb-d49e-4171-8d3c-4a15edac954d/limitador/0.log" Sep 16 14:22:16.039035 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:16.038997 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-j8p2s_e3500fa6-aac0-4404-8acb-e0452a8e6a01/kube-auth-proxy/0.log" Sep 16 14:22:17.466751 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:17.466716 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vbdpg_f8d07bb8-f953-4e1d-abce-ee0c89995cd3/main/0.log" Sep 16 14:22:17.587696 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:17.587657 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-q4x4s_e0272558-0ca2-4956-bab2-962b90a5818a/main/0.log" Sep 16 14:22:23.380489 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.380436 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xhpr/must-gather-66nsg"] Sep 16 14:22:23.380984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.380803 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd48fae1-17eb-476d-9ea5-098334261c4c" containerName="main" Sep 16 14:22:23.380984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.380814 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd48fae1-17eb-476d-9ea5-098334261c4c" containerName="main" Sep 16 14:22:23.380984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.380915 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="83422dbd-ebe3-430c-9500-18b9af62f9dc" containerName="istio-proxy" Sep 16 14:22:23.380984 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.380927 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd48fae1-17eb-476d-9ea5-098334261c4c" containerName="main" Sep 16 14:22:23.384089 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.384065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.386710 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.386686 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"kube-root-ca.crt\"" Sep 16 14:22:23.387547 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.387527 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"openshift-service-ca.crt\"" Sep 16 14:22:23.387666 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.387577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xhpr\"/\"default-dockercfg-wm7b4\"" Sep 16 14:22:23.393855 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.393829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/must-gather-66nsg"] Sep 16 14:22:23.398562 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.398531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:23.440471 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.440429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwcj\" (UniqueName: \"kubernetes.io/projected/04d78375-1068-448d-b7e8-655586eb3007-kube-api-access-5bwcj\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.440665 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.440492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04d78375-1068-448d-b7e8-655586eb3007-must-gather-output\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.541494 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.541449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwcj\" (UniqueName: \"kubernetes.io/projected/04d78375-1068-448d-b7e8-655586eb3007-kube-api-access-5bwcj\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.541494 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.541504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04d78375-1068-448d-b7e8-655586eb3007-must-gather-output\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.541805 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.541790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04d78375-1068-448d-b7e8-655586eb3007-must-gather-output\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.552080 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.552046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwcj\" (UniqueName: \"kubernetes.io/projected/04d78375-1068-448d-b7e8-655586eb3007-kube-api-access-5bwcj\") pod \"must-gather-66nsg\" (UID: \"04d78375-1068-448d-b7e8-655586eb3007\") " pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.554262 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.554236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" limit=1024 Sep 16 14:22:23.694583 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.694553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/must-gather-66nsg" Sep 16 14:22:23.966920 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.966788 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/must-gather-66nsg"] Sep 16 14:22:23.969895 ip-10-0-139-34 kubenswrapper[2559]: W0916 14:22:23.969841 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04d78375_1068_448d_b7e8_655586eb3007.slice/crio-ca9fd7e702616e95c8175e2976d6ff0779ee3280f626f8173a501dbaa2a703ae WatchSource:0}: Error finding container ca9fd7e702616e95c8175e2976d6ff0779ee3280f626f8173a501dbaa2a703ae: Status 404 returned error can't find the container with id ca9fd7e702616e95c8175e2976d6ff0779ee3280f626f8173a501dbaa2a703ae Sep 16 14:22:23.971831 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:23.971814 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:22:24.723160 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:24.723121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/must-gather-66nsg" event={"ID":"04d78375-1068-448d-b7e8-655586eb3007","Type":"ContainerStarted","Data":"ca9fd7e702616e95c8175e2976d6ff0779ee3280f626f8173a501dbaa2a703ae"} Sep 16 14:22:25.733656 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:25.733613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/must-gather-66nsg" event={"ID":"04d78375-1068-448d-b7e8-655586eb3007","Type":"ContainerStarted","Data":"71dced5be09789a8269a52970e8e22b149ff08158a72421cfd6d9b80376f6d67"} Sep 16 14:22:25.734224 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:25.733672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/must-gather-66nsg" event={"ID":"04d78375-1068-448d-b7e8-655586eb3007","Type":"ContainerStarted","Data":"11dbe0be1154b32bc8249a7a355ccad52badbfbfbb25b19b4751d7d3bc5d7095"} Sep 16 14:22:25.754727 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:25.754640 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xhpr/must-gather-66nsg" podStartSLOduration=1.792512466 podStartE2EDuration="2.754612111s" totalImagesPullingTime="962.099645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:25.75197281 +0000 UTC m=+2364.792566776" watchObservedRunningTime="2026-09-16 14:22:25.754612111 +0000 UTC m=+2364.795206067" Sep 16 14:22:26.789978 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:26.789950 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8stjr_c0af04b0-d57c-4f69-bfdb-ad22dcdfe3e0/global-pull-secret-syncer/0.log" Sep 16 14:22:26.995788 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:26.995753 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l67cb_5568d0d7-5e2c-45b5-94e3-2c412dfff0c8/konnectivity-agent/0.log" Sep 16 14:22:27.224087 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:27.224036 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-34.ec2.internal_90413d152f265712b7c74d8e704e0cb6/haproxy/0.log" Sep 16 14:22:31.850178 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:31.841254 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54f85fdc87-ccwqh_44ea1263-b681-4bfe-ae12-73e036367a72/authorino/0.log" Sep 16 14:22:31.974756 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:31.974725 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n44pr_e3d44aa3-6f60-45ce-ac22-80726fd6aa59/kuadrant-console-plugin/0.log" Sep 16 14:22:32.052109 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:32.052071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8msjx_007c82fb-d49e-4171-8d3c-4a15edac954d/limitador/0.log" Sep 16 14:22:33.594696 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.594662 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/alertmanager/0.log" Sep 16 14:22:33.654537 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.654509 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/config-reloader/0.log" Sep 16 14:22:33.695736 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.695708 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/kube-rbac-proxy-web/0.log" Sep 16 14:22:33.740946 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.740915 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/kube-rbac-proxy/0.log" Sep 16 14:22:33.795331 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.795294 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/kube-rbac-proxy-metric/0.log" Sep 16 14:22:33.827248 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.827211 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/prom-label-proxy/0.log" Sep 16 14:22:33.862048 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:33.861963 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74559091-a7b0-46e3-a115-0f5fb03f78e0/init-config-reloader/0.log" Sep 16 14:22:34.382479 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.382450 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlhzp_f6011656-7048-4f10-8b3d-66a28455f59e/node-exporter/0.log" Sep 16 14:22:34.433895 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.433851 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlhzp_f6011656-7048-4f10-8b3d-66a28455f59e/kube-rbac-proxy/0.log" Sep 16 14:22:34.499074 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.499044 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlhzp_f6011656-7048-4f10-8b3d-66a28455f59e/init-textfile/0.log" Sep 16 14:22:34.567036 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.567007 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2pjlg_5697cc5a-2e33-44dd-aacf-ebee4d46f996/kube-rbac-proxy-main/0.log" Sep 16 14:22:34.641294 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.641207 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2pjlg_5697cc5a-2e33-44dd-aacf-ebee4d46f996/kube-rbac-proxy-self/0.log" Sep 16 14:22:34.727111 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:34.727076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2pjlg_5697cc5a-2e33-44dd-aacf-ebee4d46f996/openshift-state-metrics/0.log" Sep 16 14:22:35.083710 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.083667 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kv7g8_c3869572-0c46-4889-916a-e691800e3b86/prometheus-operator/0.log" Sep 16 14:22:35.123168 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.123097 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kv7g8_c3869572-0c46-4889-916a-e691800e3b86/kube-rbac-proxy/0.log" Sep 16 14:22:35.339222 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.339141 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/thanos-query/0.log" Sep 16 14:22:35.380660 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.380628 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/kube-rbac-proxy-web/0.log" Sep 16 14:22:35.420405 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.420372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/kube-rbac-proxy/0.log" Sep 16 14:22:35.462485 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.462456 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/prom-label-proxy/0.log" Sep 16 14:22:35.507055 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.507013 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/kube-rbac-proxy-rules/0.log" Sep 16 14:22:35.544405 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:35.544371 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c87f9bd9-nrl67_a8ead6e1-f509-4287-8261-d6da0a52c463/kube-rbac-proxy-metrics/0.log" Sep 16 14:22:36.895706 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:36.895653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-w5jsw_7b204615-e64d-4ccb-9ae0-e7c8bc4a95d3/networking-console-plugin/0.log" Sep 16 14:22:37.505254 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.505208 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg"] Sep 16 14:22:37.511406 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.511363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.524078 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.523065 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg"] Sep 16 14:22:37.610917 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.610824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-podres\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.611209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.610982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-sys\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.611209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.611051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698gz\" (UniqueName: \"kubernetes.io/projected/19c11bd9-7265-4415-92d3-e9e4ae23107f-kube-api-access-698gz\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.611209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.611103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-lib-modules\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.611209 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.611132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-proc\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712502 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-sys\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698gz\" (UniqueName: \"kubernetes.io/projected/19c11bd9-7265-4415-92d3-e9e4ae23107f-kube-api-access-698gz\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-sys\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-lib-modules\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-proc\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712701 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-podres\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712727 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-lib-modules\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-proc\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.712986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.712806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19c11bd9-7265-4415-92d3-e9e4ae23107f-podres\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.727553 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.727030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-698gz\" (UniqueName: \"kubernetes.io/projected/19c11bd9-7265-4415-92d3-e9e4ae23107f-kube-api-access-698gz\") pod \"perf-node-gather-daemonset-mcwrg\" (UID: \"19c11bd9-7265-4415-92d3-e9e4ae23107f\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:37.826214 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:37.825839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:38.137108 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.137000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg"] Sep 16 14:22:38.392116 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.391970 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-rbxrh_6052c427-0ff6-48e3-a6d9-0b14d656e9ea/download-server/0.log" Sep 16 14:22:38.831375 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.831326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" event={"ID":"19c11bd9-7265-4415-92d3-e9e4ae23107f","Type":"ContainerStarted","Data":"efe952b7f6576cca999b9a71e32711b81edd491a16ecb41566149504e29189bb"} Sep 16 14:22:38.831595 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.831405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" event={"ID":"19c11bd9-7265-4415-92d3-e9e4ae23107f","Type":"ContainerStarted","Data":"f3a1ec208ae97bf49017687b6892b94ad8e06e0c12a05c3e53bc88eadc286b44"} Sep 16 14:22:38.832379 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.832349 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:38.853662 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:38.853564 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" podStartSLOduration=1.8535421909999998 podStartE2EDuration="1.853542191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:38.852409913 +0000 UTC m=+2377.893003873" watchObservedRunningTime="2026-09-16 14:22:38.853542191 +0000 UTC m=+2377.894136149" Sep 16 14:22:40.513842 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:40.513809 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvlqk_17ccbdc0-e1e0-4459-9ea8-e6b04119d22d/dns/0.log" Sep 16 14:22:40.543940 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:40.543913 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvlqk_17ccbdc0-e1e0-4459-9ea8-e6b04119d22d/kube-rbac-proxy/0.log" Sep 16 14:22:40.787626 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:40.787533 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrbdq_96d54b1b-882a-487d-b624-2840f40fb916/dns-node-resolver/0.log" Sep 16 14:22:41.460935 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:41.460898 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b5f48df84-4rfvr_ab4dbb1d-5d1a-4836-980f-06e0f3be548a/registry/0.log" Sep 16 14:22:41.494446 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:41.494369 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bhsxg_e8b9189c-5eb4-4da6-a0fa-6672a6eb69a7/node-ca/0.log" Sep 16 14:22:43.511033 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:43.510998 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-j8p2s_e3500fa6-aac0-4404-8acb-e0452a8e6a01/kube-auth-proxy/0.log" Sep 16 14:22:44.770124 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:44.770092 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xn5kz_0c830c48-2260-48df-84b1-b919795bc7c0/serve-healthcheck-canary/0.log" Sep 16 14:22:45.365337 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:45.365302 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgzsc_c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae/kube-rbac-proxy/0.log" Sep 16 14:22:45.393240 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:45.393208 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgzsc_c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae/exporter/0.log" Sep 16 14:22:45.419596 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:45.419547 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgzsc_c43e15af-2fdb-4dc5-a0dc-c9fa5f8f1eae/extractor/0.log" Sep 16 14:22:45.859986 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:45.859956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-mcwrg" Sep 16 14:22:48.705430 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:48.705378 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-j8p2s_e3500fa6-aac0-4404-8acb-e0452a8e6a01/kube-auth-proxy/0.log" Sep 16 14:22:48.891658 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:48.891618 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8h9d4_93d4bf31-20f4-4f5d-ac8e-f4378f782545/manager/0.log" Sep 16 14:22:48.983768 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:48.983671 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-7sfbx_4a92a250-a4cc-4c2f-98ca-c5aa03acc32b/manager/0.log" Sep 16 14:22:49.005121 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:49.005094 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-7sfbx_4a92a250-a4cc-4c2f-98ca-c5aa03acc32b/copy-manifests/0.log" Sep 16 14:22:50.569372 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:50.569334 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6465fbddb8-rn9vz_c8ffc9fe-8377-4047-912c-bedca466d013/manager/0.log" Sep 16 14:22:51.251388 ip-10-0-139-34 kubenswrapper[2559]: I0916 14:22:51.251348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-j8p2s_e3500fa6-aac0-4404-8acb-e0452a8e6a01/kube-auth-proxy/0.log"