Sep 09 15:49:27.764567 ip-10-0-130-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 15:49:27.764582 ip-10-0-130-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 15:49:27.764591 ip-10-0-130-57 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 15:49:27.764785 ip-10-0-130-57 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 15:49:38.013818 ip-10-0-130-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 15:49:38.013838 ip-10-0-130-57 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9e251723e0eb47748427301306d87496 -- Sep 09 15:51:19.202812 ip-10-0-130-57 systemd[1]: Starting Kubernetes Kubelet... Sep 09 15:51:19.638357 ip-10-0-130-57 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 15:51:19.638357 ip-10-0-130-57 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 15:51:19.638357 ip-10-0-130-57 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 15:51:19.638357 ip-10-0-130-57 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 15:51:19.638357 ip-10-0-130-57 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 15:51:19.640703 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.640626 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 15:51:19.645836 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645815 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.645836 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645833 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.645836 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645838 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.645836 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645842 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.645836 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645845 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645848 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645851 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645855 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645858 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645860 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645863 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645866 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645869 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645871 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645874 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645876 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645879 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645882 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645885 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645887 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645890 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645892 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645895 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.646024 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645897 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645900 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645902 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645906 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645908 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645911 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645916 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645919 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645921 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645924 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645927 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645929 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645932 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645935 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645938 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645941 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645944 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645946 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645949 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645951 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.646494 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645954 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645957 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645959 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645962 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645964 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645967 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645969 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645972 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645974 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645977 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645981 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645985 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645988 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645991 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645994 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645996 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.645999 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646004 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646007 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.647011 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646011 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646014 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646017 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646021 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646024 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646027 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646032 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646035 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646038 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646040 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646044 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646046 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646053 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646056 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646059 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646061 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646064 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646067 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646069 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646072 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.647482 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646074 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646078 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646081 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646083 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646086 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646088 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646091 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646093 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646096 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646098 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646101 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646103 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646106 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646109 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646111 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646114 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646117 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646827 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646834 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646838 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.647993 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646842 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646847 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646850 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646854 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646857 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646859 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646862 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646865 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646868 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646870 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646873 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646875 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646878 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646881 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646885 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646887 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646890 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646892 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646894 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.648483 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646897 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646901 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646904 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646907 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646910 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646913 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646916 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646919 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646922 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646924 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646928 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646931 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646933 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646936 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646940 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646943 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646946 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646948 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646951 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.648983 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646954 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646956 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646959 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646961 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646964 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646967 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646969 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646972 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646974 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646978 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646980 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646983 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646986 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646988 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646991 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646993 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646996 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.646999 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647001 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.649453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647004 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647006 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647009 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647012 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647015 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647017 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647020 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647023 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647026 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647028 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647031 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647033 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647036 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647039 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647042 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647044 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647046 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647050 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647053 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647055 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.649950 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647058 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647060 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647063 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647066 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647069 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647071 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647073 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647076 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647079 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647081 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647084 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647086 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647089 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647091 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647094 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647097 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647100 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647103 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647106 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647465 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.650456 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647470 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647472 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647475 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647477 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647480 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647483 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647486 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647488 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647491 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647494 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647497 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647500 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647503 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647505 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647507 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647526 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647529 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647532 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647535 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647537 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647540 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.650984 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647543 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647546 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647549 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647554 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647557 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647561 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647565 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647568 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647571 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647573 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647576 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647579 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647583 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647586 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647589 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647591 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647594 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647597 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.651537 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647599 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647602 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647604 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647606 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647609 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647612 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647614 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647617 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647619 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647623 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647625 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647628 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647630 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647633 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647636 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647638 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647641 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647645 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647647 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.651982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647650 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647652 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647655 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647657 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647660 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647663 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647665 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647668 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647670 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647673 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647675 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647678 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647680 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647683 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647685 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647687 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647690 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647692 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647695 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647697 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.652447 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647699 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647702 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647704 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647711 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647714 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647716 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647719 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647722 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647724 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647726 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647731 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647735 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647738 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647741 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647744 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647746 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647749 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647752 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647754 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.647756 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.652982 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647827 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647868 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647875 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647879 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647883 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647887 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647891 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647895 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647898 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647900 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647904 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647907 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647910 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647913 2562 flags.go:64] FLAG: --cgroup-root="" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647916 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647919 2562 flags.go:64] FLAG: --client-ca-file="" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647922 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647926 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647930 2562 flags.go:64] FLAG: --cluster-domain="" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647932 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647936 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647939 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647943 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647947 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 15:51:19.653484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647950 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647953 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647956 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647959 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647962 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647965 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647968 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647971 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647974 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647978 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647980 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647983 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647986 2562 flags.go:64] FLAG: --enable-server="true" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647989 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647994 2562 flags.go:64] FLAG: --event-burst="100" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.647998 2562 flags.go:64] FLAG: --event-qps="50" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648001 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648004 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648007 2562 flags.go:64] FLAG: --eviction-hard="" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648011 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648014 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648017 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648020 2562 flags.go:64] FLAG: --eviction-soft="" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648023 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648026 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 15:51:19.654085 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648031 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648035 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648038 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648041 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648043 2562 flags.go:64] FLAG: --feature-gates="" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648049 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648052 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648055 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648058 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648061 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648064 2562 flags.go:64] FLAG: --help="false" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648067 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648070 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648073 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648076 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648079 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648083 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648085 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648088 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648091 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648094 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648111 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648115 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648118 2562 flags.go:64] FLAG: --kube-reserved="" Sep 09 15:51:19.654769 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648122 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648125 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648128 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648131 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648134 2562 flags.go:64] FLAG: --lock-file="" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648137 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648140 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648144 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648151 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648154 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648157 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648160 2562 flags.go:64] FLAG: --logging-format="text" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648163 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648168 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648171 2562 flags.go:64] FLAG: --manifest-url="" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648173 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648177 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648181 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648185 2562 flags.go:64] FLAG: --max-pods="110" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648188 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648191 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648193 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648196 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648199 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648202 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 15:51:19.655370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648205 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648212 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648215 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648218 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648221 2562 flags.go:64] FLAG: --pod-cidr="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648224 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648230 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648233 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648236 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648238 2562 flags.go:64] FLAG: --port="10250" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648241 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648244 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b527c9ffe493fb24" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648247 2562 flags.go:64] FLAG: --qos-reserved="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648250 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648253 2562 flags.go:64] FLAG: --register-node="true" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648258 2562 flags.go:64] FLAG: --register-with-taints="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648261 2562 flags.go:64] FLAG: --registry-burst="10" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648264 2562 flags.go:64] FLAG: --registry-qps="5" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648267 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648270 2562 flags.go:64] FLAG: --reserved-memory="" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648273 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648278 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648280 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648283 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648286 2562 flags.go:64] FLAG: --runonce="false" Sep 09 15:51:19.656414 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648289 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648292 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648295 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648298 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648301 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648304 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648307 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648310 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648312 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648315 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648318 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648321 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648324 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648326 2562 flags.go:64] FLAG: --system-cgroups="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648329 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648334 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648337 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648340 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648344 2562 flags.go:64] FLAG: --tls-min-version="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648346 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648349 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648352 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648356 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648359 2562 flags.go:64] FLAG: --v="2" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648362 2562 flags.go:64] FLAG: --version="false" Sep 09 15:51:19.657600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648366 2562 flags.go:64] FLAG: --vmodule="" Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648370 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.648373 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648480 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648484 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648487 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648491 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648493 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648496 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648500 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648502 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648505 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648508 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648523 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648526 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648528 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648531 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648534 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648536 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648539 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.658349 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648541 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648544 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648547 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648553 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648556 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648558 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648561 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648564 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648566 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648571 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648573 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648576 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648578 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648582 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648584 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648588 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648591 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648594 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648597 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648600 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.659031 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648602 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648604 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648607 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648609 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648612 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648614 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648617 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648620 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648622 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648625 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648627 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648630 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648633 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648635 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648638 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648641 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648643 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648646 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648648 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.660033 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648652 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648655 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648660 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648663 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648665 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648668 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648671 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648674 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648678 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648681 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648683 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648686 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648689 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648692 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648694 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648697 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648699 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648702 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648705 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.660665 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648707 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648710 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648712 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648715 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648717 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648720 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648724 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648727 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648730 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648732 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648735 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648738 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648741 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648743 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648746 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648751 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648754 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648756 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648760 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.661277 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648764 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648767 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648772 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648775 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.648778 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.649598 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.657849 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.657867 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.658791 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.658805 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658913 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658921 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658927 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658931 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658936 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.662099 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658940 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658945 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658949 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658953 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658960 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658966 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658971 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658976 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658981 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658986 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658990 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658995 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.658999 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659003 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659007 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659011 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659016 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659020 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659024 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.662775 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659028 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659032 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659036 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659039 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659043 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659056 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659061 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659065 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659069 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659073 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659078 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659084 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659090 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659095 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659099 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659104 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659108 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659112 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659116 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.663363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659121 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659125 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659129 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659133 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659137 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659141 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659146 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659150 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659154 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659159 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659163 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659167 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659171 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659175 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659179 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659183 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659188 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659192 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659198 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659209 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.663891 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659214 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659218 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659222 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659226 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659230 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659234 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659238 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659242 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659246 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659251 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659255 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659261 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659266 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659270 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659274 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659278 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659282 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659286 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659290 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659295 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.664475 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659299 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659303 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659308 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659311 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659315 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659320 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659324 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659328 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659332 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659336 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659341 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659345 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659357 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659362 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659366 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.664991 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659370 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.659378 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659614 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659626 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659632 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659637 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659642 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659647 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659651 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659654 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659658 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659663 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659667 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659671 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 15:51:19.665440 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659675 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659680 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659684 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659689 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659693 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659697 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659701 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659705 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659709 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659713 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659717 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659721 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659725 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659730 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659735 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659739 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659751 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659755 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659759 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659763 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 15:51:19.665854 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659767 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659771 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659775 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659779 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659784 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659788 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659792 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659796 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659800 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659804 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659809 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659813 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659817 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659821 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659826 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659830 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659834 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659838 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659842 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659846 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 15:51:19.666363 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659850 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659854 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659858 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659864 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659871 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659875 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659879 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659884 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659888 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659902 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659907 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659911 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659915 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659919 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659923 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659928 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659932 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659936 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659940 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 15:51:19.666889 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659944 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659948 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659952 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659956 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659960 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659964 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659969 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659973 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659977 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659981 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659985 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659989 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659994 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.659998 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660002 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660006 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660010 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660015 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660018 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660023 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660027 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 15:51:19.667586 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660031 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660037 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660049 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660054 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660059 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660064 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:19.660069 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.660078 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.660240 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.663234 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.664292 2562 server.go:1019] "Starting client certificate rotation" Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.664394 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.664430 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 15:51:19.668359 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.667857 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 15:51:19.671985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.671962 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 15:51:19.683260 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.683243 2562 log.go:25] "Validated CRI v1 runtime API" Sep 09 15:51:19.689232 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.689217 2562 log.go:25] "Validated CRI v1 image API" Sep 09 15:51:19.690081 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.690063 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 15:51:19.693456 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.693441 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 15:51:19.697012 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.696990 2562 fs.go:135] Filesystem UUIDs: map[33566cda-7a84-4916-9453-39ae89706565:/dev/nvme0n1p3 64c6c6c8-aded-40c8-a8a9-8f42f867db5a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 09 15:51:19.697083 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.697012 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 15:51:19.701367 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.701266 2562 manager.go:216] Machine: {Timestamp:2026-09-09 15:51:19.700802736 +0000 UTC m=+0.379889269 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107736 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a4ac6cb8f765cded9d5c649baf42 SystemUUID:ec29a4ac-6cb8-f765-cded-9d5c649baf42 BootID:9e251723-e0eb-4774-8427-301306d87496 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dc:80:e9:54:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dc:80:e9:54:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:3f:19:25:fc:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 15:51:19.701367 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.701363 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 15:51:19.701467 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.701427 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 15:51:19.704874 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.704852 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 15:51:19.704997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.704878 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 15:51:19.705688 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.705678 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 15:51:19.705723 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.705691 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 15:51:19.705723 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.705703 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 15:51:19.706493 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.706483 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 15:51:19.707698 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.707688 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 15:51:19.711704 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711693 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 09 15:51:19.711839 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711829 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 15:51:19.711874 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711858 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 09 15:51:19.711874 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711868 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 15:51:19.711930 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711879 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 15:51:19.711930 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711888 2562 kubelet.go:398] "Adding apiserver pod source" Sep 09 15:51:19.711930 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.711896 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 15:51:19.712230 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.712209 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 15:51:19.712325 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.712236 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 15:51:19.714498 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.714483 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 15:51:19.717785 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.717771 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 15:51:19.717848 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.717799 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 15:51:19.719969 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.719956 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.719976 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.719982 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.719988 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.719997 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720002 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720015 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 15:51:19.720023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720020 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720029 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720034 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720048 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720053 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720077 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 15:51:19.720204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.720082 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 15:51:19.723169 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.723047 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 15:51:19.723781 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.723761 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 15:51:19.723781 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.723760 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 15:51:19.723907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.723791 2562 server.go:1268] "Started kubelet" Sep 09 15:51:19.723907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.723858 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 15:51:19.723907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.723888 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 15:51:19.724014 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.723928 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 15:51:19.724639 ip-10-0-130-57 systemd[1]: Started Kubernetes Kubelet. Sep 09 15:51:19.727833 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.727811 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 15:51:19.730002 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.729905 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 09 15:51:19.734738 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.733087 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 15:51:19.735462 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.734432 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18d3b1e64f17adda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:ip-10-0-130-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-09-09 15:51:19.72376521 +0000 UTC m=+0.402851763,LastTimestamp:2026-09-09 15:51:19.72376521 +0000 UTC m=+0.402851763,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}" Sep 09 15:51:19.736836 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.736816 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 15:51:19.737400 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.737385 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 15:51:19.737524 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.737490 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:19.738026 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.737999 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 15:51:19.738026 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738020 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 15:51:19.738157 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738002 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 15:51:19.738157 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738132 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 09 15:51:19.738157 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738138 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 09 15:51:19.738265 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738166 2562 factory.go:55] Registering systemd factory Sep 09 15:51:19.738265 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738181 2562 factory.go:223] Registration of the systemd container factory successfully Sep 09 15:51:19.738397 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738383 2562 factory.go:153] Registering CRI-O factory Sep 09 15:51:19.738397 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738396 2562 factory.go:223] Registration of the crio container factory successfully Sep 09 15:51:19.738486 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738440 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 15:51:19.738486 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738454 2562 factory.go:103] Registering Raw factory Sep 09 15:51:19.738486 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738465 2562 manager.go:1196] Started watching for new ooms in manager Sep 09 15:51:19.738874 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.738862 2562 manager.go:349] Starting recovery of all containers Sep 09 15:51:19.742424 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.742390 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 15:51:19.742555 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.742532 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 15:51:19.743963 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.743938 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27v57" Sep 09 15:51:19.748361 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.748346 2562 manager.go:354] Recovery completed Sep 09 15:51:19.749642 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.749622 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27v57" Sep 09 15:51:19.753366 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.753321 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.755177 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755161 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.755261 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755192 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.755261 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755203 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.755750 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755737 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 15:51:19.755803 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755750 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 15:51:19.755803 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755767 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 09 15:51:19.755803 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.755750 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.756685 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.756671 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.756749 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.756696 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.756749 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.756706 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.757173 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.757083 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18d3b1e650f6f5fc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:ip-10-0-130-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-09-09 15:51:19.75517542 +0000 UTC m=+0.434261953,LastTimestamp:2026-09-09 15:51:19.75517542 +0000 UTC m=+0.434261953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}" Sep 09 15:51:19.757376 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.757364 2562 policy_none.go:49] "None policy: Start" Sep 09 15:51:19.757417 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.757380 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 15:51:19.757417 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.757392 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 15:51:19.759135 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.759120 2562 policy_none.go:47] "Start" Sep 09 15:51:19.804097 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804081 2562 manager.go:337] "Starting Device Plugin manager" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.804119 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804129 2562 server.go:87] "Starting device plugin registration server" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804420 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804430 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804564 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804654 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.804664 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.805122 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 15:51:19.820363 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.805165 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:19.843718 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.843664 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 15:51:19.844840 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.844820 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 15:51:19.844908 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.844871 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 15:51:19.844908 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.844894 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 15:51:19.845018 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.844937 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 15:51:19.847021 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.847000 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:19.905677 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.905610 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.906413 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.906398 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.906496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.906426 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.906496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.906437 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.906496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.906457 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.915129 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.915105 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.915214 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.915129 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-57.ec2.internal\": node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:19.941390 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.941370 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:19.945576 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.945552 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"] Sep 09 15:51:19.945627 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.945621 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.946869 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.946849 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.946947 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.946877 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.946947 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.946887 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.949271 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949259 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.949398 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.949435 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949414 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.949941 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949926 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.949941 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949936 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.950080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949951 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.950080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949957 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.950080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949966 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.950080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.949970 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.952244 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.952230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.952322 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.952255 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 15:51:19.952969 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.952953 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory" Sep 09 15:51:19.953058 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.952980 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 15:51:19.953058 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:19.952993 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID" Sep 09 15:51:19.967032 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.967014 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal" Sep 09 15:51:19.970375 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:19.970361 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.040037 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.040017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.040114 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.040041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.040114 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.040058 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b59b194d8dabcdf6b93ea8511f43438d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"b59b194d8dabcdf6b93ea8511f43438d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.042121 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:20.042107 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:20.140537 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.140537 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.140660 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b59b194d8dabcdf6b93ea8511f43438d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"b59b194d8dabcdf6b93ea8511f43438d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.140660 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.140660 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b59b194d8dabcdf6b93ea8511f43438d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"b59b194d8dabcdf6b93ea8511f43438d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.140660 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.140638 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e1a1bd51f2b5db3df0d1181edc45308-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"6e1a1bd51f2b5db3df0d1181edc45308\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.142609 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:20.142596 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:20.243466 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:20.243392 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:20.268574 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.268550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.273465 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:20.273296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:20.344173 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:20.344142 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:21.782524 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:21.782499 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:21.882553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:21.882481 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:21.882679 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:21.882577 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:21.983102 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:21.983081 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:22.002235 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.002215 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 15:51:22.002348 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.002332 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 15:51:22.002383 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.002369 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 15:51:22.083738 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:22.083719 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found" Sep 09 15:51:22.092384 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.092351 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 15:46:21 +0000 UTC" deadline="2028-05-03 01:03:34.417063651 +0000 UTC" Sep 09 15:51:22.092448 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.092382 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14433h12m12.324684512s" Sep 09 15:51:22.136568 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.136511 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:22.175804 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.175782 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" Sep 09 15:51:22.186530 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.186513 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 15:51:22.187320 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.187308 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" Sep 09 15:51:22.198315 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.198301 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 15:51:22.396498 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.396436 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:22.487102 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:22.487073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59b194d8dabcdf6b93ea8511f43438d.slice/crio-3000ee91d8b4f819083ee8b09520d67de0b75116346aae5c757792f14c3fffc2 WatchSource:0}: Error finding container 3000ee91d8b4f819083ee8b09520d67de0b75116346aae5c757792f14c3fffc2: Status 404 returned error can't find the container with id 3000ee91d8b4f819083ee8b09520d67de0b75116346aae5c757792f14c3fffc2 Sep 09 15:51:22.492927 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:22.492804 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 15:51:23.006025 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.005993 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:23.051033 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.050993 2562 apiserver.go:52] "Watching apiserver" Sep 09 15:51:23.060111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.060087 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 15:51:23.060836 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.060806 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","openshift-multus/multus-q2vlw","openshift-network-operator/iptables-alerter-5h7ll","kube-system/global-pull-secret-syncer-t5z2s","openshift-cluster-node-tuning-operator/tuned-4c4rv","openshift-image-registry/node-ca-z6m4g","openshift-multus/multus-additional-cni-plugins-2vnvh","openshift-multus/network-metrics-daemon-wpwhx","openshift-network-diagnostics/network-check-target-8kgr4","openshift-ovn-kubernetes/ovnkube-node-vfn9r","kube-system/konnectivity-agent-rqfdc","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt","openshift-dns/node-resolver-nt5bs"] Sep 09 15:51:23.063333 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.063313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.065557 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.065534 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.065903 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.065874 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 15:51:23.066130 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.066110 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 15:51:23.066200 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.066139 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 15:51:23.066309 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.066252 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6hdm9\"" Sep 09 15:51:23.066365 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.066311 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.066433 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.066362 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.067998 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.067816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.067998 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.067853 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.068280 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.068260 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dsnwk\"" Sep 09 15:51:23.068648 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.068472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 15:51:23.068648 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.068622 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.070286 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.070035 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.070286 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.070103 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:23.070286 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.070114 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.070494 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.070400 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 15:51:23.070642 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.070625 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.070810 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.070731 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2qjpd\"" Sep 09 15:51:23.072622 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.072605 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.075013 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.074709 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.075213 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.075195 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 15:51:23.075998 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.075979 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rs6q7\"" Sep 09 15:51:23.076446 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.076428 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.076540 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.076520 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.077048 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.077024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:23.077149 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.077122 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:23.077149 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.077132 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x2swg\"" Sep 09 15:51:23.077304 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.077174 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.077525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.077496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.077525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.077502 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 15:51:23.080302 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.079284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.081924 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.081616 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.083162 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.082858 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ssmlc\"" Sep 09 15:51:23.083162 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.082874 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.083162 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.082944 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.084487 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.084468 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.084685 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.084666 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 15:51:23.084886 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.084870 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xlh9x\"" Sep 09 15:51:23.086587 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-kubernetes\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-sys\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086663 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-host\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdm2h\" (UniqueName: \"kubernetes.io/projected/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kube-api-access-hdm2h\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-dbus\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b746718-910e-48d2-b23c-4f25fab41d05-iptables-alerter-script\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysconfig\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-systemd\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-tuned\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-sys-fs\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.086925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-system-cni-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.086961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b746718-910e-48d2-b23c-4f25fab41d05-host-slash\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9n7\" (UniqueName: \"kubernetes.io/projected/4b746718-910e-48d2-b23c-4f25fab41d05-kube-api-access-wf9n7\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-tmp\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-registration-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-device-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087110 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087137 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-var-lib-kubelet\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-serviceca\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pr8h\" (UniqueName: \"kubernetes.io/projected/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-kube-api-access-2pr8h\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.087297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-socket-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxlv\" (UniqueName: \"kubernetes.io/projected/82d45f58-1805-4c8d-978e-09aab955a0a8-kube-api-access-flxlv\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-conf\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-run\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mdn\" (UniqueName: \"kubernetes.io/projected/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-kube-api-access-c7mdn\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-lib-modules\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-host\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.087899 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.088329 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-kubelet-config\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.088329 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-cnibin\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.088329 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.087998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-modprobe-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.088329 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.088262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 15:51:23.088918 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.088897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 15:51:23.089002 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.088986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 15:51:23.089125 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.089108 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9pdd7\"" Sep 09 15:51:23.089539 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.089520 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.089753 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.089738 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.090301 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.089878 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 15:51:23.090301 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.089822 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:23.092714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.092696 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 15:51:23.093683 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.093663 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 15:51:23.094648 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.094627 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dnnjn\"" Sep 09 15:51:23.094726 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.094710 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 15:51:23.094789 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.094627 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 15:51:23.112195 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.112175 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tjj8" Sep 09 15:51:23.120680 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.120658 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tjj8" Sep 09 15:51:23.176631 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.176601 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 15:51:23.187937 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.187883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"b59b194d8dabcdf6b93ea8511f43438d","Type":"ContainerStarted","Data":"3000ee91d8b4f819083ee8b09520d67de0b75116346aae5c757792f14c3fffc2"} Sep 09 15:51:23.189137 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdm2h\" (UniqueName: \"kubernetes.io/projected/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kube-api-access-hdm2h\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.189269 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b746718-910e-48d2-b23c-4f25fab41d05-iptables-alerter-script\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.189269 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-ovn\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.189269 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-bin\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.189269 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-multus-daemon-config\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.189269 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-multus-certs\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189273 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-etc-kubernetes\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-systemd\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-tuned\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-slash\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-etc-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-systemd\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.189525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b783ea01-682c-4b85-adea-0c3cff2e3734-hosts-file\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-sys-fs\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b746718-910e-48d2-b23c-4f25fab41d05-host-slash\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9n7\" (UniqueName: \"kubernetes.io/projected/4b746718-910e-48d2-b23c-4f25fab41d05-kube-api-access-wf9n7\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-sys-fs\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7pb\" (UniqueName: \"kubernetes.io/projected/b783ea01-682c-4b85-adea-0c3cff2e3734-kube-api-access-sx7pb\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-var-lib-kubelet\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-serviceca\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.189826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-log-socket\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b783ea01-682c-4b85-adea-0c3cff2e3734-tmp-dir\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b746718-910e-48d2-b23c-4f25fab41d05-iptables-alerter-script\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189860 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-multus\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-system-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-var-lib-kubelet\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.189959 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-socket-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxlv\" (UniqueName: \"kubernetes.io/projected/82d45f58-1805-4c8d-978e-09aab955a0a8-kube-api-access-flxlv\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b746718-910e-48d2-b23c-4f25fab41d05-host-slash\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-run\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mdn\" (UniqueName: \"kubernetes.io/projected/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-kube-api-access-c7mdn\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-socket-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-netns\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.190507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-serviceca\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-cni-binary-copy\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfd2t\" (UniqueName: \"kubernetes.io/projected/16db5106-107a-491f-95f0-0b2012d6deeb-kube-api-access-cfd2t\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-run\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-lib-modules\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovn-node-metrics-cert\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-socket-dir-parent\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-cnibin\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-kubernetes\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-netd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-cnibin\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190620 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-script-lib\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-lib-modules\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s856n\" (UniqueName: \"kubernetes.io/projected/4dc49812-c7e5-4525-85ed-01be2448c4d0-kube-api-access-s856n\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-kubernetes\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.191296 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-hostroot\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-dbus\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-kubelet\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysconfig\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-dbus\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-var-lib-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190901 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysconfig\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-system-cni-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190940 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82d45f58-1805-4c8d-978e-09aab955a0a8-system-cni-dir\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-tmp\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.190980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-systemd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cxl\" (UniqueName: \"kubernetes.io/projected/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-kube-api-access-c5cxl\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-netns\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-registration-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-device-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.192208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/881e116f-eb06-4df6-87a9-2cdf700fdd72-agent-certs\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-kubelet\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-registration-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pr8h\" (UniqueName: \"kubernetes.io/projected/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-kube-api-access-2pr8h\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-env-overrides\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-k8s-cni-cncf-io\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-device-dir\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-bin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-conf-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.191384 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-conf\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-host\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.191515 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:23.6914825 +0000 UTC m=+3.032674323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:23.193111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-host\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-systemd-units\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-node-log\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-config\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-conf\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-cnibin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-kubelet-config\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-modprobe-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04238531-1ed9-461f-ac8d-d46b0af52bb2-kubelet-config\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.191976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-modprobe-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-sys\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-sysctl-d\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192119 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-sys\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.193934 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-host\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.194798 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192186 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82d45f58-1805-4c8d-978e-09aab955a0a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.194798 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.194798 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-host\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.194798 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.192256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/881e116f-eb06-4df6-87a9-2cdf700fdd72-konnectivity-ca\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.195733 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.195710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-etc-tuned\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.195915 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.195893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-tmp\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.200613 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.200475 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:23.200613 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.200501 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:23.200613 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.200523 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:23.200613 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.200586 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:23.700564297 +0000 UTC m=+3.041756107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:23.203928 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.203900 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mdn\" (UniqueName: \"kubernetes.io/projected/7f6ea669-ea32-4fbb-8337-85e2e00a0a56-kube-api-access-c7mdn\") pod \"tuned-4c4rv\" (UID: \"7f6ea669-ea32-4fbb-8337-85e2e00a0a56\") " pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.204375 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.204325 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxlv\" (UniqueName: \"kubernetes.io/projected/82d45f58-1805-4c8d-978e-09aab955a0a8-kube-api-access-flxlv\") pod \"multus-additional-cni-plugins-2vnvh\" (UID: \"82d45f58-1805-4c8d-978e-09aab955a0a8\") " pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.205742 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.205716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdm2h\" (UniqueName: \"kubernetes.io/projected/09f04bef-e70a-4c9e-8eb3-a9106d0a49b1-kube-api-access-hdm2h\") pod \"aws-ebs-csi-driver-node-55htt\" (UID: \"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.206838 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.206816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9n7\" (UniqueName: \"kubernetes.io/projected/4b746718-910e-48d2-b23c-4f25fab41d05-kube-api-access-wf9n7\") pod \"iptables-alerter-5h7ll\" (UID: \"4b746718-910e-48d2-b23c-4f25fab41d05\") " pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.207332 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.207307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pr8h\" (UniqueName: \"kubernetes.io/projected/65c46cff-df1c-4afe-9ee8-a8e48413f0c1-kube-api-access-2pr8h\") pod \"node-ca-z6m4g\" (UID: \"65c46cff-df1c-4afe-9ee8-a8e48413f0c1\") " pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.293100 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-var-lib-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293100 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-systemd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293100 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cxl\" (UniqueName: \"kubernetes.io/projected/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-kube-api-access-c5cxl\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-netns\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/881e116f-eb06-4df6-87a9-2cdf700fdd72-agent-certs\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-var-lib-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-kubelet\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-systemd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-env-overrides\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-kubelet\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-netns\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-k8s-cni-cncf-io\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293294 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-bin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-conf-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-systemd-units\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-node-log\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-config\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-cnibin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.293767 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-bin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293515 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-cnibin\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-systemd-units\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-node-log\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/881e116f-eb06-4df6-87a9-2cdf700fdd72-konnectivity-ca\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-ovn\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-bin\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-run-ovn\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293611 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-conf-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-multus-daemon-config\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-multus-certs\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293691 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-bin\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-k8s-cni-cncf-io\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-run-multus-certs\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-etc-kubernetes\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-slash\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.294623 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-etc-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b783ea01-682c-4b85-adea-0c3cff2e3734-hosts-file\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293864 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-slash\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7pb\" (UniqueName: \"kubernetes.io/projected/b783ea01-682c-4b85-adea-0c3cff2e3734-kube-api-access-sx7pb\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-etc-kubernetes\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-log-socket\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b783ea01-682c-4b85-adea-0c3cff2e3734-tmp-dir\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-env-overrides\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-multus\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.293988 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.293996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-system-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-netns\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-etc-openvswitch\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-cni-binary-copy\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfd2t\" (UniqueName: \"kubernetes.io/projected/16db5106-107a-491f-95f0-0b2012d6deeb-kube-api-access-cfd2t\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294083 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b783ea01-682c-4b85-adea-0c3cff2e3734-hosts-file\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.295475 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.294094 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:23.794076739 +0000 UTC m=+3.135268564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/881e116f-eb06-4df6-87a9-2cdf700fdd72-konnectivity-ca\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-multus-daemon-config\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-host-var-lib-cni-multus\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovn-node-metrics-cert\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-socket-dir-parent\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b783ea01-682c-4b85-adea-0c3cff2e3734-tmp-dir\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-netd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-run-netns\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-system-cni-dir\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-script-lib\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-log-socket\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-multus-socket-dir-parent\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s856n\" (UniqueName: \"kubernetes.io/projected/4dc49812-c7e5-4525-85ed-01be2448c4d0-kube-api-access-s856n\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-cni-netd\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-hostroot\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-kubelet\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.296403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-config\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294665 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16db5106-107a-491f-95f0-0b2012d6deeb-hostroot\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-kubelet\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dc49812-c7e5-4525-85ed-01be2448c4d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16db5106-107a-491f-95f0-0b2012d6deeb-cni-binary-copy\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.294914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovnkube-script-lib\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.296128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/881e116f-eb06-4df6-87a9-2cdf700fdd72-agent-certs\") pod \"konnectivity-agent-rqfdc\" (UID: \"881e116f-eb06-4df6-87a9-2cdf700fdd72\") " pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.297152 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.297045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dc49812-c7e5-4525-85ed-01be2448c4d0-ovn-node-metrics-cert\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.297526 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:23.297257 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1a1bd51f2b5db3df0d1181edc45308.slice/crio-23ce4c8555a2a13ed70176e4a10908a969de5bf39cb75f68132d90641e451834 WatchSource:0}: Error finding container 23ce4c8555a2a13ed70176e4a10908a969de5bf39cb75f68132d90641e451834: Status 404 returned error can't find the container with id 23ce4c8555a2a13ed70176e4a10908a969de5bf39cb75f68132d90641e451834 Sep 09 15:51:23.302509 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.302460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cxl\" (UniqueName: \"kubernetes.io/projected/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-kube-api-access-c5cxl\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.303249 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.303226 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfd2t\" (UniqueName: \"kubernetes.io/projected/16db5106-107a-491f-95f0-0b2012d6deeb-kube-api-access-cfd2t\") pod \"multus-q2vlw\" (UID: \"16db5106-107a-491f-95f0-0b2012d6deeb\") " pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.303704 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.303656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7pb\" (UniqueName: \"kubernetes.io/projected/b783ea01-682c-4b85-adea-0c3cff2e3734-kube-api-access-sx7pb\") pod \"node-resolver-nt5bs\" (UID: \"b783ea01-682c-4b85-adea-0c3cff2e3734\") " pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.303980 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.303959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s856n\" (UniqueName: \"kubernetes.io/projected/4dc49812-c7e5-4525-85ed-01be2448c4d0-kube-api-access-s856n\") pod \"ovnkube-node-vfn9r\" (UID: \"4dc49812-c7e5-4525-85ed-01be2448c4d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.375905 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.375875 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" Sep 09 15:51:23.384749 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.384731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" Sep 09 15:51:23.397501 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.397469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5h7ll" Sep 09 15:51:23.403503 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.403483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" Sep 09 15:51:23.411050 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.411031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6m4g" Sep 09 15:51:23.418549 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.418531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nt5bs" Sep 09 15:51:23.426008 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.425987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q2vlw" Sep 09 15:51:23.433477 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.433459 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:23.439048 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.439032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:23.697908 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.697831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:23.698062 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.697986 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:23.698062 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.698053 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:24.698038412 +0000 UTC m=+4.039230235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:23.798586 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.798385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:23.798586 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.798472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:23.798586 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798559 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:23.798854 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798622 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:23.798854 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798643 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:23.798854 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798650 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:24.798630571 +0000 UTC m=+4.139822395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:23.798854 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798655 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:23.798854 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:23.798705 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:24.798690661 +0000 UTC m=+4.139882480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:23.912595 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:23.912565 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 15:51:24.027629 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.027602 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc49812_c7e5_4525_85ed_01be2448c4d0.slice/crio-33c9935617e2242c0997ffee1ab51f280c9924b7f39f66c2d43528965f927d7e WatchSource:0}: Error finding container 33c9935617e2242c0997ffee1ab51f280c9924b7f39f66c2d43528965f927d7e: Status 404 returned error can't find the container with id 33c9935617e2242c0997ffee1ab51f280c9924b7f39f66c2d43528965f927d7e Sep 09 15:51:24.028622 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.028597 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f04bef_e70a_4c9e_8eb3_a9106d0a49b1.slice/crio-93c2522df7a715ff5d3c3bf55ba09c82db03cb8a5d67b48f1b113e1241b6b98a WatchSource:0}: Error finding container 93c2522df7a715ff5d3c3bf55ba09c82db03cb8a5d67b48f1b113e1241b6b98a: Status 404 returned error can't find the container with id 93c2522df7a715ff5d3c3bf55ba09c82db03cb8a5d67b48f1b113e1241b6b98a Sep 09 15:51:24.032018 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.031995 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb783ea01_682c_4b85_adea_0c3cff2e3734.slice/crio-8977ea6ccecc3f4ad35a73cdeb6dd4aedccaeb6fa156a759a26d6576c12b9af2 WatchSource:0}: Error finding container 8977ea6ccecc3f4ad35a73cdeb6dd4aedccaeb6fa156a759a26d6576c12b9af2: Status 404 returned error can't find the container with id 8977ea6ccecc3f4ad35a73cdeb6dd4aedccaeb6fa156a759a26d6576c12b9af2 Sep 09 15:51:24.034981 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.034929 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16db5106_107a_491f_95f0_0b2012d6deeb.slice/crio-38ea43c10da5d39371be402b7c008f14d4c6255b51ca19f3c81c6b54a6b2e737 WatchSource:0}: Error finding container 38ea43c10da5d39371be402b7c008f14d4c6255b51ca19f3c81c6b54a6b2e737: Status 404 returned error can't find the container with id 38ea43c10da5d39371be402b7c008f14d4c6255b51ca19f3c81c6b54a6b2e737 Sep 09 15:51:24.036318 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.036274 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881e116f_eb06_4df6_87a9_2cdf700fdd72.slice/crio-46e11bc9386100c9532af6027cd466151a0bceba2ba8219736fc87ca6b11dcb7 WatchSource:0}: Error finding container 46e11bc9386100c9532af6027cd466151a0bceba2ba8219736fc87ca6b11dcb7: Status 404 returned error can't find the container with id 46e11bc9386100c9532af6027cd466151a0bceba2ba8219736fc87ca6b11dcb7 Sep 09 15:51:24.037028 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.036935 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6ea669_ea32_4fbb_8337_85e2e00a0a56.slice/crio-9d499c7c74bd1e18e286a991212b6c3f387af028b60661d6c7e64bad61e0197e WatchSource:0}: Error finding container 9d499c7c74bd1e18e286a991212b6c3f387af028b60661d6c7e64bad61e0197e: Status 404 returned error can't find the container with id 9d499c7c74bd1e18e286a991212b6c3f387af028b60661d6c7e64bad61e0197e Sep 09 15:51:24.059438 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.059386 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b746718_910e_48d2_b23c_4f25fab41d05.slice/crio-766e51190cfea60890ee216c6f1171abe8855749a303214401ed0a996e1197b2 WatchSource:0}: Error finding container 766e51190cfea60890ee216c6f1171abe8855749a303214401ed0a996e1197b2: Status 404 returned error can't find the container with id 766e51190cfea60890ee216c6f1171abe8855749a303214401ed0a996e1197b2 Sep 09 15:51:24.060131 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.060109 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c46cff_df1c_4afe_9ee8_a8e48413f0c1.slice/crio-6b31847dd42ec53b153821e61396fa2831406f4cfe7722dd03d7ba7ffa029f4f WatchSource:0}: Error finding container 6b31847dd42ec53b153821e61396fa2831406f4cfe7722dd03d7ba7ffa029f4f: Status 404 returned error can't find the container with id 6b31847dd42ec53b153821e61396fa2831406f4cfe7722dd03d7ba7ffa029f4f Sep 09 15:51:24.061251 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:24.061228 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d45f58_1805_4c8d_978e_09aab955a0a8.slice/crio-f372c3e471ce1886ed9a9082c2093f6e784d25afe905cd8b572a43ba83c80f9e WatchSource:0}: Error finding container f372c3e471ce1886ed9a9082c2093f6e784d25afe905cd8b572a43ba83c80f9e: Status 404 returned error can't find the container with id f372c3e471ce1886ed9a9082c2093f6e784d25afe905cd8b572a43ba83c80f9e Sep 09 15:51:24.124363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.124321 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:46:23 +0000 UTC" deadline="2028-02-07 07:06:44.959839837 +0000 UTC" Sep 09 15:51:24.124363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.124358 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12375h15m20.835484222s" Sep 09 15:51:24.191243 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.191208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"b59b194d8dabcdf6b93ea8511f43438d","Type":"ContainerStarted","Data":"9a05ab3301e2c7812a3539e2c051db75273feb3e125562bf7f6d277fef0e5e2a"} Sep 09 15:51:24.192480 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.192453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5h7ll" event={"ID":"4b746718-910e-48d2-b23c-4f25fab41d05","Type":"ContainerStarted","Data":"766e51190cfea60890ee216c6f1171abe8855749a303214401ed0a996e1197b2"} Sep 09 15:51:24.193516 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.193496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q2vlw" event={"ID":"16db5106-107a-491f-95f0-0b2012d6deeb","Type":"ContainerStarted","Data":"38ea43c10da5d39371be402b7c008f14d4c6255b51ca19f3c81c6b54a6b2e737"} Sep 09 15:51:24.194557 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.194538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" event={"ID":"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1","Type":"ContainerStarted","Data":"93c2522df7a715ff5d3c3bf55ba09c82db03cb8a5d67b48f1b113e1241b6b98a"} Sep 09 15:51:24.195651 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.195630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"33c9935617e2242c0997ffee1ab51f280c9924b7f39f66c2d43528965f927d7e"} Sep 09 15:51:24.196583 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.196557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerStarted","Data":"f372c3e471ce1886ed9a9082c2093f6e784d25afe905cd8b572a43ba83c80f9e"} Sep 09 15:51:24.197489 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.197471 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6m4g" event={"ID":"65c46cff-df1c-4afe-9ee8-a8e48413f0c1","Type":"ContainerStarted","Data":"6b31847dd42ec53b153821e61396fa2831406f4cfe7722dd03d7ba7ffa029f4f"} Sep 09 15:51:24.198455 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.198435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" event={"ID":"7f6ea669-ea32-4fbb-8337-85e2e00a0a56","Type":"ContainerStarted","Data":"9d499c7c74bd1e18e286a991212b6c3f387af028b60661d6c7e64bad61e0197e"} Sep 09 15:51:24.199559 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.199538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqfdc" event={"ID":"881e116f-eb06-4df6-87a9-2cdf700fdd72","Type":"ContainerStarted","Data":"46e11bc9386100c9532af6027cd466151a0bceba2ba8219736fc87ca6b11dcb7"} Sep 09 15:51:24.200637 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.200619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nt5bs" event={"ID":"b783ea01-682c-4b85-adea-0c3cff2e3734","Type":"ContainerStarted","Data":"8977ea6ccecc3f4ad35a73cdeb6dd4aedccaeb6fa156a759a26d6576c12b9af2"} Sep 09 15:51:24.201613 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.201595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"6e1a1bd51f2b5db3df0d1181edc45308","Type":"ContainerStarted","Data":"23ce4c8555a2a13ed70176e4a10908a969de5bf39cb75f68132d90641e451834"} Sep 09 15:51:24.205378 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.205330 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" podStartSLOduration=2.205317974 podStartE2EDuration="2.205317974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:24.204680668 +0000 UTC m=+3.545872494" watchObservedRunningTime="2026-09-09 15:51:24.205317974 +0000 UTC m=+3.546509803" Sep 09 15:51:24.704190 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.704153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:24.704384 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.704294 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:24.704384 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.704354 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:26.704336302 +0000 UTC m=+6.045528115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:24.804942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.804902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:24.805124 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:24.804972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:24.805191 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805149 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:24.805191 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805173 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:24.805191 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805186 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:24.805343 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805248 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:26.805229496 +0000 UTC m=+6.146421326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:24.805343 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805332 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:24.805476 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:24.805369 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:26.805356942 +0000 UTC m=+6.146548767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:25.124784 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:25.124665 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:46:23 +0000 UTC" deadline="2028-05-21 23:26:42.196187496 +0000 UTC" Sep 09 15:51:25.124784 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:25.124707 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14887h35m17.071486085s" Sep 09 15:51:25.185963 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:25.185935 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:25.186129 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:25.186065 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:25.186458 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:25.186436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:25.186593 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:25.186553 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:25.186593 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:25.186588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:25.186696 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:25.186672 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:26.720597 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:26.720555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:26.721170 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.720699 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:26.721170 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.720772 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:30.720752204 +0000 UTC m=+10.061944034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:26.821472 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:26.821435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:26.821667 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:26.821529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:26.821808 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.821667 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:26.821808 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.821727 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:30.82170964 +0000 UTC m=+10.162901465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:26.822139 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.822119 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:26.822221 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.822145 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:26.822221 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.822158 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:26.822221 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:26.822202 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:30.822186832 +0000 UTC m=+10.163378642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:27.183617 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:27.183483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:27.183617 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:27.183483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:27.183617 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:27.183483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:27.183873 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:27.183621 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:27.183873 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:27.183720 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:27.183873 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:27.183796 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:29.183179 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:29.183149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:29.183662 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:29.183275 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:29.183739 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:29.183719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:29.183847 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:29.183824 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:29.183985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:29.183968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:29.184082 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:29.184052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:30.753952 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:30.753918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:30.754305 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.754070 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:30.754305 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.754133 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.754114914 +0000 UTC m=+18.095306735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:30.854732 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:30.854689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:30.854917 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:30.854790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:30.854917 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.854856 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:30.854917 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.854882 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:30.855058 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.854936 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.854915726 +0000 UTC m=+18.196107546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:30.855058 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.854883 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:30.855058 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.854972 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:30.855058 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:30.855027 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:38.855011184 +0000 UTC m=+18.196203008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:31.189037 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.189010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:31.189546 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:31.189521 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:31.189738 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.189263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:31.190091 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.189231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:31.190091 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:31.190062 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:31.190313 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:31.189920 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:31.221908 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.221808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" event={"ID":"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1","Type":"ContainerStarted","Data":"046e78605862c90a1e9f76d6fd1e5ee96a7d312fbc5a1006eb940274201b5ce8"} Sep 09 15:51:31.225234 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.225175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6m4g" event={"ID":"65c46cff-df1c-4afe-9ee8-a8e48413f0c1","Type":"ContainerStarted","Data":"41d53e0aaf7a9f9d936541cdf9a6da8bc43073f26dd1f33308205c89ad40e2d7"} Sep 09 15:51:31.228805 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.228241 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqfdc" event={"ID":"881e116f-eb06-4df6-87a9-2cdf700fdd72","Type":"ContainerStarted","Data":"df7ee18a9ae59247133c85535270deb32bf5f353cd0b52dab7f96b420b2fffa4"} Sep 09 15:51:31.233985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.233910 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e1a1bd51f2b5db3df0d1181edc45308" containerID="f66cd7238c205b6a405f1d6138916c9828371f3a69e2c6508e5d581c8ebf6c51" exitCode=0 Sep 09 15:51:31.233985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.233947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"6e1a1bd51f2b5db3df0d1181edc45308","Type":"ContainerDied","Data":"f66cd7238c205b6a405f1d6138916c9828371f3a69e2c6508e5d581c8ebf6c51"} Sep 09 15:51:31.242669 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.242622 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6m4g" podStartSLOduration=3.603936907 podStartE2EDuration="10.242606141s" totalImagesPullingTime="6.638669234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:31.241786749 +0000 UTC m=+10.582978579" watchObservedRunningTime="2026-09-09 15:51:31.242606141 +0000 UTC m=+10.583797972" Sep 09 15:51:31.262261 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:31.261011 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rqfdc" podStartSLOduration=3.619356336 podStartE2EDuration="10.260991676s" totalImagesPullingTime="6.64163534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:31.260755587 +0000 UTC m=+10.601947416" watchObservedRunningTime="2026-09-09 15:51:31.260991676 +0000 UTC m=+10.602183507" Sep 09 15:51:32.239127 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.239092 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nt5bs" event={"ID":"b783ea01-682c-4b85-adea-0c3cff2e3734","Type":"ContainerStarted","Data":"f33d6ca9be5af860b79a59e78532aac0550e1c05f95bb2b4db12f39b0838965f"} Sep 09 15:51:32.242583 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.242556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"6e1a1bd51f2b5db3df0d1181edc45308","Type":"ContainerStarted","Data":"4a7ace4326db28b4436933caf4fb5aaa237e266f8c534606c4225785afbbc552"} Sep 09 15:51:32.245903 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.245840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5h7ll" event={"ID":"4b746718-910e-48d2-b23c-4f25fab41d05","Type":"ContainerStarted","Data":"af41f9e19f8121d6bed8b1d0074a55bb75321cdc8fe2a949208097093bb53476"} Sep 09 15:51:32.275063 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.274506 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nt5bs" podStartSLOduration=4.205674604 podStartE2EDuration="11.274485822s" totalImagesPullingTime="7.068811218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:32.2581319 +0000 UTC m=+11.599323728" watchObservedRunningTime="2026-09-09 15:51:32.274485822 +0000 UTC m=+11.615677646" Sep 09 15:51:32.288232 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.288110 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" podStartSLOduration=10.288091995 podStartE2EDuration="10.288091995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:32.28769141 +0000 UTC m=+11.628883294" watchObservedRunningTime="2026-09-09 15:51:32.288091995 +0000 UTC m=+11.629283823" Sep 09 15:51:32.288366 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:32.288279 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5h7ll" podStartSLOduration=4.635143869 podStartE2EDuration="11.288272458s" totalImagesPullingTime="6.653128589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:32.274962338 +0000 UTC m=+11.616154169" watchObservedRunningTime="2026-09-09 15:51:32.288272458 +0000 UTC m=+11.629464300" Sep 09 15:51:33.183533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.183496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:33.183693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.183496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:33.183693 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:33.183640 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:33.183776 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:33.183691 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:33.183776 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.183496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:33.183856 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:33.183781 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:33.249053 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.249010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerStarted","Data":"6ea2714f5dcaa1b05781b015636ee4e9b023c60478c5ca02505e0eef5aa2d696"} Sep 09 15:51:33.436496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.436399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:33.437102 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:33.437083 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:35.183917 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:35.183850 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:35.183917 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:35.183876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:35.183917 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:35.183899 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:35.184347 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:35.183983 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:35.184347 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:35.184121 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:35.184347 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:35.184196 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:35.254993 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:35.254960 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="6ea2714f5dcaa1b05781b015636ee4e9b023c60478c5ca02505e0eef5aa2d696" exitCode=0 Sep 09 15:51:35.255169 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:35.255013 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"6ea2714f5dcaa1b05781b015636ee4e9b023c60478c5ca02505e0eef5aa2d696"} Sep 09 15:51:37.183854 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:37.183815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:37.183854 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:37.183853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:37.184384 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:37.183815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:37.184384 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:37.183942 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:37.184384 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:37.183998 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:37.184384 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:37.184075 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:38.817711 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:38.817508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:38.817711 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.817703 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:38.818253 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.817785 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:54.817764113 +0000 UTC m=+34.158955935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:38.918707 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:38.918658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:38.918888 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:38.918754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:38.918888 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.918862 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:38.919007 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.918922 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:54.91890442 +0000 UTC m=+34.260096245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:38.919007 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.918860 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:38.919007 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.918968 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:38.919007 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.918981 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:38.919225 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:38.919049 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:51:54.919032443 +0000 UTC m=+34.260224267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:39.183861 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:39.183828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:39.183861 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:39.183862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:39.184174 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:39.183828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:39.184174 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:39.183955 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:39.184174 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:39.184030 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:39.184174 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:39.184106 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:41.184156 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:41.184127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:41.184644 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:41.184318 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:41.184644 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:41.184256 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:41.184644 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:41.184446 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:41.184644 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:41.184471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:41.184644 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:41.184554 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:43.183283 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:43.183249 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:43.183762 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:43.183249 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:43.183762 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:43.183395 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:43.183762 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:43.183467 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:43.183762 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:43.183257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:43.183762 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:43.183588 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:44.243795 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.243769 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 15:51:44.275075 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.275036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" event={"ID":"7f6ea669-ea32-4fbb-8337-85e2e00a0a56","Type":"ContainerStarted","Data":"ea2ab559872c6984ce9033aac606c71262c42d7acba96f6779647ec3bb6ecae2"} Sep 09 15:51:44.276671 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.276642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q2vlw" event={"ID":"16db5106-107a-491f-95f0-0b2012d6deeb","Type":"ContainerStarted","Data":"4650a40dadca93fb11f637bf85e459f826bddcb93507b47c3fd2b2fe6233bbc4"} Sep 09 15:51:44.278520 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.278497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" event={"ID":"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1","Type":"ContainerStarted","Data":"1b492255f0482d1ee5835e7cfc9c77669fa477f2e22540565548a8bb0845d284"} Sep 09 15:51:44.279995 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.279974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"0202adb688606dc3d3df56b2a186efd1a056f6d26e04cdbbeff6ea1c970e77ed"} Sep 09 15:51:44.290804 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:44.290764 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4c4rv" podStartSLOduration=3.287268949 podStartE2EDuration="23.290750037s" totalImagesPullingTime="20.003481088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:44.290167967 +0000 UTC m=+23.631359795" watchObservedRunningTime="2026-09-09 15:51:44.290750037 +0000 UTC m=+23.631941866" Sep 09 15:51:45.158235 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.157981 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T15:51:44.243788954Z","UUID":"af27c531-e7ea-488c-aaf7-1f8a3603c4c3","Handler":null,"Name":"","Endpoint":""} Sep 09 15:51:45.159543 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.159527 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 15:51:45.159635 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.159554 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 15:51:45.183076 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.183049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:45.183245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.183092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:45.183245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.183092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:45.183245 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:45.183232 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:45.184070 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:45.184020 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:45.184324 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:45.184301 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:45.284960 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.284925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"e5d41ef311b68c99727073865f4213c6db4db978449abb4b77900c695a560a04"} Sep 09 15:51:45.285358 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.284999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"bf8ac87db10bfe7858679d9be62cf202fc56bfd2539e4965908f5711fe49a89b"} Sep 09 15:51:45.285358 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.285031 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"cb81aa5e27a3fa6172df4169b2bfa3e02f3bfe1835244528a05052ef0dfa51a9"} Sep 09 15:51:45.285358 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.285057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"94ed30d5fd0a5b142765a80c4faabd7ed3d4d4a45f18596d634ee72f11b16542"} Sep 09 15:51:45.285358 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.285084 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"eb417cfbdc665827e7f3163405036fbfd5298b978d5d221aa4f85c652896b113"} Sep 09 15:51:45.286742 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.286721 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="989607d1b12d6b26a2d7d1c7efbdf2a205845bcc6d29e6518240e7887b12ac32" exitCode=0 Sep 09 15:51:45.286838 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.286799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"989607d1b12d6b26a2d7d1c7efbdf2a205845bcc6d29e6518240e7887b12ac32"} Sep 09 15:51:45.305992 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:45.305949 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q2vlw" podStartSLOduration=4.302934568 podStartE2EDuration="24.305934433s" totalImagesPullingTime="20.002999865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:44.304511486 +0000 UTC m=+23.645703318" watchObservedRunningTime="2026-09-09 15:51:45.305934433 +0000 UTC m=+24.647126253" Sep 09 15:51:46.290706 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:46.290480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" event={"ID":"09f04bef-e70a-4c9e-8eb3-a9106d0a49b1","Type":"ContainerStarted","Data":"89fed0ba9208c8253dbb538e6528f5ea4e8aa5bd6eae9705f5943af016dd7d4d"} Sep 09 15:51:46.307146 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:46.307100 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55htt" podStartSLOduration=4.32078771 podStartE2EDuration="25.307085294s" totalImagesPullingTime="20.986297584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:46.305600782 +0000 UTC m=+25.646792610" watchObservedRunningTime="2026-09-09 15:51:46.307085294 +0000 UTC m=+25.648277123" Sep 09 15:51:47.183185 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.183157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:47.183365 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.183158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:47.183365 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:47.183262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:47.183365 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.183160 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:47.183365 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:47.183337 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:47.183539 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:47.183438 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:47.295782 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.295743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"07326ef134a2110081feece8dceaea93bb3664f0bffc12a796efa91664d3d44f"} Sep 09 15:51:47.297783 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.297758 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="4b223a9633f107e9daa2bbfdcd4bfefec3934bd886053a3c12c0439ba84a50b8" exitCode=0 Sep 09 15:51:47.297895 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:47.297826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"4b223a9633f107e9daa2bbfdcd4bfefec3934bd886053a3c12c0439ba84a50b8"} Sep 09 15:51:48.302110 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:48.302080 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="06f843448d3f2357753694137ee039180b43e7f52a5b32618207a2e3a404bd0c" exitCode=0 Sep 09 15:51:48.302514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:48.302147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"06f843448d3f2357753694137ee039180b43e7f52a5b32618207a2e3a404bd0c"} Sep 09 15:51:49.183594 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.183508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:49.183745 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.183508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:49.183745 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:49.183631 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:49.183745 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.183508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:49.183888 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:49.183712 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:49.183888 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:49.183832 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:49.309364 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.309331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" event={"ID":"4dc49812-c7e5-4525-85ed-01be2448c4d0","Type":"ContainerStarted","Data":"33fa06c1f0c6964b95ed1f88f8cb38128fc4bea20a6b7c5c19d37d2253484940"} Sep 09 15:51:49.310133 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.309678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:49.310133 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.309739 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:49.310133 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.309786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:49.327141 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.327119 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:49.327489 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.327464 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:51:49.356258 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:49.356211 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" podStartSLOduration=8.324651715 podStartE2EDuration="28.356196798s" totalImagesPullingTime="20.031545083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:49.354441483 +0000 UTC m=+28.695633311" watchObservedRunningTime="2026-09-09 15:51:49.356196798 +0000 UTC m=+28.697388627" Sep 09 15:51:50.076287 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.076118 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:50.076480 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.076448 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 15:51:50.076903 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.076874 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqfdc" Sep 09 15:51:50.870140 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.870107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8kgr4"] Sep 09 15:51:50.870600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.870269 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:50.870600 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:50.870360 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:50.873750 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.873723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5z2s"] Sep 09 15:51:50.873877 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.873856 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:50.873978 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:50.873958 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:50.874444 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.874419 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpwhx"] Sep 09 15:51:50.874538 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:50.874522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:50.874647 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:50.874624 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:52.183515 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:52.183485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:52.183515 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:52.183517 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:52.183938 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:52.183485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:52.183938 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:52.183584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:52.183938 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:52.183659 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:52.183938 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:52.183718 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:54.183634 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.183602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:54.184158 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.183610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:54.184158 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.183731 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:54.184158 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.183611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:54.184158 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.183819 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:54.184158 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.183890 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:54.830997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.830946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:54.831146 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.831115 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:54.831254 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.831188 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret podName:04238531-1ed9-461f-ac8d-d46b0af52bb2 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:26.831167175 +0000 UTC m=+66.172358999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret") pod "global-pull-secret-syncer-t5z2s" (UID: "04238531-1ed9-461f-ac8d-d46b0af52bb2") : object "kube-system"/"original-pull-secret" not registered Sep 09 15:51:54.931462 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.931436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:54.931549 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:54.931481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:54.931598 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931578 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:54.931641 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931600 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 15:51:54.931641 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931615 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 15:51:54.931641 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931625 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pzvrb for pod openshift-network-diagnostics/network-check-target-8kgr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:54.931734 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931646 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:52:26.931629973 +0000 UTC m=+66.272821783 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 15:51:54.931734 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:54.931665 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb podName:dd51558a-dfb7-4a0a-b9b6-0b69958d841e nodeName:}" failed. No retries permitted until 2026-09-09 15:52:26.931654105 +0000 UTC m=+66.272845916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzvrb" (UniqueName: "kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb") pod "network-check-target-8kgr4" (UID: "dd51558a-dfb7-4a0a-b9b6-0b69958d841e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 15:51:55.325435 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:55.325391 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="1e3beeb7aa7d9b348b1886c2cbea9d787a481eb09d065c5682956e6c30f7fe6c" exitCode=0 Sep 09 15:51:55.325786 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:55.325439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"1e3beeb7aa7d9b348b1886c2cbea9d787a481eb09d065c5682956e6c30f7fe6c"} Sep 09 15:51:56.184071 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:56.183910 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:56.184210 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:56.183947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:56.184256 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:56.183950 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:56.184256 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:56.184215 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5z2s" podUID="04238531-1ed9-461f-ac8d-d46b0af52bb2" Sep 09 15:51:56.184326 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:56.184291 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:51:56.184326 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:56.184137 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8kgr4" podUID="dd51558a-dfb7-4a0a-b9b6-0b69958d841e" Sep 09 15:51:56.330063 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:56.330029 2562 generic.go:356] "Generic (PLEG): container finished" podID="82d45f58-1805-4c8d-978e-09aab955a0a8" containerID="3773ab67b425647dd74a5cceca7b3dc1af6f6fb7d46bbdd1f13f021d1fe06527" exitCode=0 Sep 09 15:51:56.330421 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:56.330079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerDied","Data":"3773ab67b425647dd74a5cceca7b3dc1af6f6fb7d46bbdd1f13f021d1fe06527"} Sep 09 15:51:57.335884 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.335852 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" event={"ID":"82d45f58-1805-4c8d-978e-09aab955a0a8","Type":"ContainerStarted","Data":"3a6093edc8e2b7591db0bece0f94ae6a610410dfba780c0f0400b9ed938f06a1"} Sep 09 15:51:57.356840 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.356792 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2vnvh" podStartSLOduration=10.161648099 podStartE2EDuration="36.35677171s" totalImagesPullingTime="26.195123611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:57.356210479 +0000 UTC m=+36.697402308" watchObservedRunningTime="2026-09-09 15:51:57.35677171 +0000 UTC m=+36.697963540" Sep 09 15:51:57.924695 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.924670 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeReady" Sep 09 15:51:57.924860 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.924795 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 15:51:57.955876 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.955846 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:51:57.990569 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.990537 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-d7kfz"] Sep 09 15:51:57.990741 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.990699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:57.993691 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.993669 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 15:51:57.993804 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.993786 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 15:51:58.001313 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.998571 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 15:51:58.001313 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:57.999119 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wwhjq\"" Sep 09 15:51:58.004149 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.003782 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 15:51:58.004316 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.004295 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-d7kfz"] Sep 09 15:51:58.004378 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.004324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:51:58.004378 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.004341 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-srs75"] Sep 09 15:51:58.004507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.004434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.007393 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.007367 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mzxcf\"" Sep 09 15:51:58.007519 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.007402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 15:51:58.007519 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.007402 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 15:51:58.020604 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.020582 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wbgbv"] Sep 09 15:51:58.020710 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.020700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.023348 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.023326 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 15:51:58.023564 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.023549 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p7tqp\"" Sep 09 15:51:58.023654 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.023552 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 15:51:58.023800 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.023786 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 15:51:58.035370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.035351 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srs75"] Sep 09 15:51:58.035469 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.035378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbgbv"] Sep 09 15:51:58.035510 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.035472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.038011 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.037993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5xfw9\"" Sep 09 15:51:58.038108 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.038026 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 15:51:58.038108 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.038026 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 15:51:58.052000 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.051974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052090 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052090 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052090 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcft\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052090 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052229 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052229 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.052229 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.052208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153057 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-cert\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.153057 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.153245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c40bf4af-9876-408d-9527-5115d436f397-nginx-conf\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.153245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcft\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153318 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9d60a4-51da-4d45-b1fa-64a1d251965f-config-volume\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d60a4-51da-4d45-b1fa-64a1d251965f-tmp-dir\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9857r\" (UniqueName: \"kubernetes.io/projected/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-kube-api-access-9857r\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.153385 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.153440 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:51:58.153479 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.153530 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:51:58.653511591 +0000 UTC m=+37.994703421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrz5\" (UniqueName: \"kubernetes.io/projected/ef9d60a4-51da-4d45-b1fa-64a1d251965f-kube-api-access-zhrz5\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.153897 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.153895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.154305 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.154282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.157038 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.157016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.157122 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.157054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.164165 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.164140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.164258 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.164148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcft\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.183053 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.182991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:51:58.183157 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.182991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:51:58.183204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.182991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:51:58.186038 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2dhs\"" Sep 09 15:51:58.186217 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186039 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 15:51:58.186217 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186057 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 15:51:58.186217 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186072 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 15:51:58.186217 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186073 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tvtml\"" Sep 09 15:51:58.186217 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.186131 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 15:51:58.254533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254497 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-cert\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.254533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c40bf4af-9876-408d-9527-5115d436f397-nginx-conf\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9d60a4-51da-4d45-b1fa-64a1d251965f-config-volume\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d60a4-51da-4d45-b1fa-64a1d251965f-tmp-dir\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9857r\" (UniqueName: \"kubernetes.io/projected/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-kube-api-access-9857r\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.254673 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:51:58.254765 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.254758 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:58.754736398 +0000 UTC m=+38.095928229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:51:58.255049 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrz5\" (UniqueName: \"kubernetes.io/projected/ef9d60a4-51da-4d45-b1fa-64a1d251965f-kube-api-access-zhrz5\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.255049 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.254895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.255049 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.255009 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:51:58.255049 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.255043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d60a4-51da-4d45-b1fa-64a1d251965f-tmp-dir\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.255197 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.255052 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:58.755039005 +0000 UTC m=+38.096230811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:51:58.255317 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.255297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9d60a4-51da-4d45-b1fa-64a1d251965f-config-volume\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.255354 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.255339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c40bf4af-9876-408d-9527-5115d436f397-nginx-conf\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.257114 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.257093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-cert\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.264927 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.264903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrz5\" (UniqueName: \"kubernetes.io/projected/ef9d60a4-51da-4d45-b1fa-64a1d251965f-kube-api-access-zhrz5\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.265036 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.264962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9857r\" (UniqueName: \"kubernetes.io/projected/9bcb88f1-3276-4896-bbe6-e6b29f66d4a1-kube-api-access-9857r\") pod \"ingress-canary-srs75\" (UID: \"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1\") " pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.329596 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.329559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srs75" Sep 09 15:51:58.658486 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.658448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:58.659257 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.658595 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:51:58.659257 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.658609 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:51:58.659257 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.658673 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:51:59.658654913 +0000 UTC m=+38.999846742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:51:58.660883 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.660856 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srs75"] Sep 09 15:51:58.664453 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:51:58.664425 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcb88f1_3276_4896_bbe6_e6b29f66d4a1.slice/crio-4720be49ad3d275c7add046476d6bd1d412bcd9668eb941c77644f8a03cd8ca1 WatchSource:0}: Error finding container 4720be49ad3d275c7add046476d6bd1d412bcd9668eb941c77644f8a03cd8ca1: Status 404 returned error can't find the container with id 4720be49ad3d275c7add046476d6bd1d412bcd9668eb941c77644f8a03cd8ca1 Sep 09 15:51:58.759289 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.759203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:58.759456 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:58.759328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:58.759456 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.759366 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:51:58.759456 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.759432 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:51:58.759561 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.759456 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:51:59.759436321 +0000 UTC m=+39.100628153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:51:58.759561 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:58.759486 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:51:59.759476243 +0000 UTC m=+39.100668057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:51:59.345194 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:59.345160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srs75" event={"ID":"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1","Type":"ContainerStarted","Data":"4720be49ad3d275c7add046476d6bd1d412bcd9668eb941c77644f8a03cd8ca1"} Sep 09 15:51:59.666772 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:59.666671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:51:59.667195 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.666836 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:51:59.667195 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.666855 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:51:59.667195 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.666920 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:52:01.666900117 +0000 UTC m=+41.008091949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:51:59.768060 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:59.768020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:51:59.768250 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.768147 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:51:59.768250 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:51:59.768148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:51:59.768250 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.768217 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:52:01.768198937 +0000 UTC m=+41.109390762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:51:59.768250 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.768218 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:51:59.768444 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:51:59.768306 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:01.768290956 +0000 UTC m=+41.109482778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:52:01.683400 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:01.683361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:52:01.683783 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.683514 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:52:01.683783 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.683533 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:52:01.683783 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.683586 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:52:05.683571237 +0000 UTC m=+45.024763047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:52:01.784599 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:01.784557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:52:01.784800 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:01.784690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:52:01.784800 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.784720 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:52:01.784800 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.784794 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:52:05.784777901 +0000 UTC m=+45.125969850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:52:01.784800 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.784800 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:52:01.784988 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:01.784853 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:05.784837867 +0000 UTC m=+45.126029678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:52:02.355856 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:02.355820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srs75" event={"ID":"9bcb88f1-3276-4896-bbe6-e6b29f66d4a1","Type":"ContainerStarted","Data":"fab44f0d5385ff5e2f25b2c59b43ad2f77dd45bed5768c88ec7c7f0b99a1cae6"} Sep 09 15:52:02.370506 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:02.370454 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-srs75" podStartSLOduration=2.480220977 podStartE2EDuration="5.37043796s" totalImagesPullingTime="2.890216983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:02.370143904 +0000 UTC m=+41.711335740" watchObservedRunningTime="2026-09-09 15:52:02.37043796 +0000 UTC m=+41.711629786" Sep 09 15:52:05.713644 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:05.713602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:52:05.714116 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.713789 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:52:05.714116 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.713816 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:52:05.714116 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.713901 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:52:13.71387822 +0000 UTC m=+53.055070029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:52:05.814089 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:05.814050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:52:05.814240 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:05.814097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:52:05.814240 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.814196 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:52:05.814240 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.814200 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:52:05.814334 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.814254 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:52:13.814240731 +0000 UTC m=+53.155432538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:52:05.814334 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:05.814267 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:13.814261262 +0000 UTC m=+53.155453069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:52:13.770549 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:13.770510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:52:13.770999 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.770652 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:52:13.770999 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.770670 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:52:13.770999 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.770726 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:52:29.770708608 +0000 UTC m=+69.111900418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:52:13.871776 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:13.871734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:52:13.871776 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:13.871784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:52:13.871981 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.871885 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:52:13.871981 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.871950 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:52:29.871933862 +0000 UTC m=+69.213125673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:52:13.871981 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.871892 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:52:13.872140 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:13.872054 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:52:29.872038558 +0000 UTC m=+69.213230378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:52:21.324565 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:21.324535 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfn9r" Sep 09 15:52:26.870174 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.870118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:52:26.873209 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.873190 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 15:52:26.883378 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.883352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04238531-1ed9-461f-ac8d-d46b0af52bb2-original-pull-secret\") pod \"global-pull-secret-syncer-t5z2s\" (UID: \"04238531-1ed9-461f-ac8d-d46b0af52bb2\") " pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:52:26.971203 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.971168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:52:26.971395 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.971248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:52:26.973924 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.973901 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 15:52:26.974028 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.973937 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 15:52:26.981996 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:26.981974 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 15:52:26.982088 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:26.982042 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:53:30.982021409 +0000 UTC m=+130.323213233 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : secret "metrics-daemon-secret" not found Sep 09 15:52:26.984162 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.984148 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 15:52:26.994475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:26.994449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvrb\" (UniqueName: \"kubernetes.io/projected/dd51558a-dfb7-4a0a-b9b6-0b69958d841e-kube-api-access-pzvrb\") pod \"network-check-target-8kgr4\" (UID: \"dd51558a-dfb7-4a0a-b9b6-0b69958d841e\") " pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:52:27.003241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.003223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5z2s" Sep 09 15:52:27.267771 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.267700 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5z2s"] Sep 09 15:52:27.270903 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:52:27.270877 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04238531_1ed9_461f_ac8d_d46b0af52bb2.slice/crio-238160dd2b3961bcb9a4010feb53134276332d3f8ce8a0617205099f261c19e5 WatchSource:0}: Error finding container 238160dd2b3961bcb9a4010feb53134276332d3f8ce8a0617205099f261c19e5: Status 404 returned error can't find the container with id 238160dd2b3961bcb9a4010feb53134276332d3f8ce8a0617205099f261c19e5 Sep 09 15:52:27.294982 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.294959 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tvtml\"" Sep 09 15:52:27.302955 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.302930 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:52:27.412248 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.412209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5z2s" event={"ID":"04238531-1ed9-461f-ac8d-d46b0af52bb2","Type":"ContainerStarted","Data":"238160dd2b3961bcb9a4010feb53134276332d3f8ce8a0617205099f261c19e5"} Sep 09 15:52:27.566768 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:27.566695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8kgr4"] Sep 09 15:52:27.569588 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:52:27.569560 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd51558a_dfb7_4a0a_b9b6_0b69958d841e.slice/crio-813114e4ada13af7ea74f4741a022113dcf26bbe547b27c291e5b44946c6bfc6 WatchSource:0}: Error finding container 813114e4ada13af7ea74f4741a022113dcf26bbe547b27c291e5b44946c6bfc6: Status 404 returned error can't find the container with id 813114e4ada13af7ea74f4741a022113dcf26bbe547b27c291e5b44946c6bfc6 Sep 09 15:52:28.416449 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:28.416392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8kgr4" event={"ID":"dd51558a-dfb7-4a0a-b9b6-0b69958d841e","Type":"ContainerStarted","Data":"813114e4ada13af7ea74f4741a022113dcf26bbe547b27c291e5b44946c6bfc6"} Sep 09 15:52:29.791613 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:29.791571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:52:29.792181 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.791754 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:52:29.792181 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.791778 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:52:29.792181 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.791862 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:53:01.791839213 +0000 UTC m=+101.133031036 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:52:29.892837 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:29.892795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:52:29.893031 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:29.892852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:52:29.893031 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.892962 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:52:29.893031 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.892964 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:52:29.893031 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.893025 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:53:01.893005915 +0000 UTC m=+101.234197742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:52:29.893224 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:52:29.893043 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:53:01.893033698 +0000 UTC m=+101.234225506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:52:32.429302 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:32.429259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5z2s" event={"ID":"04238531-1ed9-461f-ac8d-d46b0af52bb2","Type":"ContainerStarted","Data":"0cc53e2761293869352cb14aac8fe0c9f1a215a39c3edac743f57ebe4537d62d"} Sep 09 15:52:32.430896 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:32.430866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8kgr4" event={"ID":"dd51558a-dfb7-4a0a-b9b6-0b69958d841e","Type":"ContainerStarted","Data":"dccfd1e09d1bdd7dec2fc118e31ebe61c50c8acda2a2e0632677de2d2aff15ff"} Sep 09 15:52:32.431006 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:32.430989 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:52:32.444131 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:32.444088 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t5z2s" podStartSLOduration=66.687729901 podStartE2EDuration="1m11.444075587s" totalImagesPullingTime="4.756345686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:32.443497915 +0000 UTC m=+71.784689745" watchObservedRunningTime="2026-09-09 15:52:32.444075587 +0000 UTC m=+71.785267469" Sep 09 15:52:32.457723 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:52:32.457685 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8kgr4" podStartSLOduration=67.004242156 podStartE2EDuration="1m11.457672719s" totalImagesPullingTime="4.453430563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:52:32.456723522 +0000 UTC m=+71.797915351" watchObservedRunningTime="2026-09-09 15:52:32.457672719 +0000 UTC m=+71.798864548" Sep 09 15:53:01.822237 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:53:01.822180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:53:01.822698 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.822341 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:53:01.822698 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.822359 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:53:01.822698 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.822439 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:54:05.822400753 +0000 UTC m=+165.163592574 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:53:01.923269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:53:01.923398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.923758 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.923928 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:54:05.923902164 +0000 UTC m=+165.265093991 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.924514 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:53:01.925931 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:01.924582 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:54:05.924565897 +0000 UTC m=+165.265757722 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:53:03.435787 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:53:03.435754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8kgr4" Sep 09 15:53:31.024977 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:53:31.024943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:53:31.025465 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:31.025051 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 15:53:31.025465 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:53:31.025102 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:55:33.025089826 +0000 UTC m=+252.366281632 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : secret "metrics-daemon-secret" not found Sep 09 15:54:01.007348 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:01.007294 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" Sep 09 15:54:01.013561 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:01.013531 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" podUID="c40bf4af-9876-408d-9527-5115d436f397" Sep 09 15:54:01.042883 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:01.042850 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wbgbv" podUID="ef9d60a4-51da-4d45-b1fa-64a1d251965f" Sep 09 15:54:01.198727 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:01.198691 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:54:01.638057 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:01.638026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:54:05.871356 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:05.871314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:54:05.871839 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.871441 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 15:54:05.871839 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.871454 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5494898f59-lqwlc: secret "image-registry-tls" not found Sep 09 15:54:05.871839 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.871512 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls podName:79f9c5e2-bb0b-498d-82db-a203c5a1feea nodeName:}" failed. No retries permitted until 2026-09-09 15:56:07.871499088 +0000 UTC m=+287.212690895 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls") pod "image-registry-5494898f59-lqwlc" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea") : secret "image-registry-tls" not found Sep 09 15:54:05.972079 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:05.972044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:54:05.972182 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:05.972092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:54:05.972252 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.972228 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 15:54:05.972303 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.972267 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 15:54:05.972358 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.972319 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert podName:c40bf4af-9876-408d-9527-5115d436f397 nodeName:}" failed. No retries permitted until 2026-09-09 15:56:07.972295611 +0000 UTC m=+287.313487432 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-d7kfz" (UID: "c40bf4af-9876-408d-9527-5115d436f397") : secret "networking-console-plugin-cert" not found Sep 09 15:54:05.972358 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:54:05.972341 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls podName:ef9d60a4-51da-4d45-b1fa-64a1d251965f nodeName:}" failed. No retries permitted until 2026-09-09 15:56:07.972330916 +0000 UTC m=+287.313522739 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls") pod "dns-default-wbgbv" (UID: "ef9d60a4-51da-4d45-b1fa-64a1d251965f") : secret "dns-default-metrics-tls" not found Sep 09 15:54:12.183553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:12.183499 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:54:15.183439 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:15.183348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbgbv" Sep 09 15:54:16.183213 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:54:16.183182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:55:33.093262 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:33.093211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:55:33.093790 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:33.093363 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 15:55:33.093790 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:33.093441 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs podName:337bfd9d-63ad-43b9-9b6f-e79aab96a48f nodeName:}" failed. No retries permitted until 2026-09-09 15:57:35.093424722 +0000 UTC m=+374.434616542 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs") pod "network-metrics-daemon-wpwhx" (UID: "337bfd9d-63ad-43b9-9b6f-e79aab96a48f") : secret "metrics-daemon-secret" not found Sep 09 15:55:38.688331 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.688285 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp"] Sep 09 15:55:38.690856 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.690832 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl"] Sep 09 15:55:38.691397 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.691365 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.693484 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.693463 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq"] Sep 09 15:55:38.693583 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.693559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" Sep 09 15:55:38.695716 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695694 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 15:55:38.695890 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695865 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 15:55:38.695997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695920 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 15:55:38.695997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qzrff\"" Sep 09 15:55:38.695997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695869 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 15:55:38.695997 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.695903 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:38.697179 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.697159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:55:38.697258 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.697163 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-h9cz7\"" Sep 09 15:55:38.697258 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.697209 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 15:55:38.698983 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.698631 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 15:55:38.698983 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.698731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 15:55:38.698983 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.698880 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:55:38.699448 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.699429 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hht9b\"" Sep 09 15:55:38.705708 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.705645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp"] Sep 09 15:55:38.706570 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.706523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl"] Sep 09 15:55:38.707423 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.707390 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq"] Sep 09 15:55:38.773869 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.773843 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gt8ms"] Sep 09 15:55:38.776526 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.776507 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" Sep 09 15:55:38.777659 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.777443 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t"] Sep 09 15:55:38.779209 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.779193 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9jcqk\"" Sep 09 15:55:38.779915 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.779896 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rl7db"] Sep 09 15:55:38.780022 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.780009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:38.782344 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.782324 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 15:55:38.782524 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.782509 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:38.782603 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.782589 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 15:55:38.782711 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.782591 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:55:38.782886 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.782864 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 15:55:38.783026 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.783008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sjjrf\"" Sep 09 15:55:38.785128 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.785112 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 15:55:38.785311 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.785293 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 15:55:38.786337 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.786315 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 15:55:38.786513 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.786476 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 15:55:38.786601 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.786545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k2p6g\"" Sep 09 15:55:38.789401 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.789383 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gt8ms"] Sep 09 15:55:38.795330 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.795307 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 15:55:38.796024 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.795999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t"] Sep 09 15:55:38.798338 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.798317 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rl7db"] Sep 09 15:55:38.833694 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zwr\" (UniqueName: \"kubernetes.io/projected/7c3aaeaa-985d-4ea4-a623-2522319a941a-kube-api-access-l9zwr\") pod \"volume-data-source-validator-695c895bd6-rjthl\" (UID: \"7c3aaeaa-985d-4ea4-a623-2522319a941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" Sep 09 15:55:38.833878 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:38.833878 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.833985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.833985 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjd9\" (UniqueName: \"kubernetes.io/projected/11c3ccaa-3771-4320-8342-1e4bf1c839c2-kube-api-access-5kjd9\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:38.834064 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.833991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-kube-api-access-wbhvv\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.935277 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjd9\" (UniqueName: \"kubernetes.io/projected/11c3ccaa-3771-4320-8342-1e4bf1c839c2-kube-api-access-5kjd9\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:38.935277 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935281 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-serving-cert\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:38.935514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zwr\" (UniqueName: \"kubernetes.io/projected/7c3aaeaa-985d-4ea4-a623-2522319a941a-kube-api-access-l9zwr\") pod \"volume-data-source-validator-695c895bd6-rjthl\" (UID: \"7c3aaeaa-985d-4ea4-a623-2522319a941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" Sep 09 15:55:38.935514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4g48\" (UniqueName: \"kubernetes.io/projected/d26dd5f9-339d-4123-b5e7-bf490e802699-kube-api-access-g4g48\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:38.935514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:38.935514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-config\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-trusted-ca\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-config\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:38.935596 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:38.935657 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls podName:11c3ccaa-3771-4320-8342-1e4bf1c839c2 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:39.435637278 +0000 UTC m=+258.776829101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cx7tq" (UID: "11c3ccaa-3771-4320-8342-1e4bf1c839c2") : secret "samples-operator-tls" not found Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:38.935658 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:38.935714 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4g5n\" (UniqueName: \"kubernetes.io/projected/1eeafd31-c7ce-4911-98bd-773a62b87ba5-kube-api-access-r4g5n\") pod \"network-check-source-685456655f-gt8ms\" (UID: \"1eeafd31-c7ce-4911-98bd-773a62b87ba5\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" Sep 09 15:55:38.936121 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-kube-api-access-wbhvv\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.936121 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:38.935777 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:39.435751538 +0000 UTC m=+258.776943359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:38.936121 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d26dd5f9-339d-4123-b5e7-bf490e802699-serving-cert\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:38.936121 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.935845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8p4\" (UniqueName: \"kubernetes.io/projected/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-kube-api-access-tg8p4\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:38.936337 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.936302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.943907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.943853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-kube-api-access-wbhvv\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:38.943907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.943874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zwr\" (UniqueName: \"kubernetes.io/projected/7c3aaeaa-985d-4ea4-a623-2522319a941a-kube-api-access-l9zwr\") pod \"volume-data-source-validator-695c895bd6-rjthl\" (UID: \"7c3aaeaa-985d-4ea4-a623-2522319a941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" Sep 09 15:55:38.943907 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:38.943855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjd9\" (UniqueName: \"kubernetes.io/projected/11c3ccaa-3771-4320-8342-1e4bf1c839c2-kube-api-access-5kjd9\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:39.008817 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.008784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" Sep 09 15:55:39.036563 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-serving-cert\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.036680 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4g48\" (UniqueName: \"kubernetes.io/projected/d26dd5f9-339d-4123-b5e7-bf490e802699-kube-api-access-g4g48\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.036733 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-config\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.036733 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-trusted-ca\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.036828 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-config\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.036828 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4g5n\" (UniqueName: \"kubernetes.io/projected/1eeafd31-c7ce-4911-98bd-773a62b87ba5-kube-api-access-r4g5n\") pod \"network-check-source-685456655f-gt8ms\" (UID: \"1eeafd31-c7ce-4911-98bd-773a62b87ba5\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" Sep 09 15:55:39.036921 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d26dd5f9-339d-4123-b5e7-bf490e802699-serving-cert\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.036921 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.036877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8p4\" (UniqueName: \"kubernetes.io/projected/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-kube-api-access-tg8p4\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.037365 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.037342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-config\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.037523 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.037499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-config\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.037838 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.037816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d26dd5f9-339d-4123-b5e7-bf490e802699-trusted-ca\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.039038 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.039019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d26dd5f9-339d-4123-b5e7-bf490e802699-serving-cert\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.039403 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.039384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-serving-cert\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.045473 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.045432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4g5n\" (UniqueName: \"kubernetes.io/projected/1eeafd31-c7ce-4911-98bd-773a62b87ba5-kube-api-access-r4g5n\") pod \"network-check-source-685456655f-gt8ms\" (UID: \"1eeafd31-c7ce-4911-98bd-773a62b87ba5\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" Sep 09 15:55:39.045574 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.045472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4g48\" (UniqueName: \"kubernetes.io/projected/d26dd5f9-339d-4123-b5e7-bf490e802699-kube-api-access-g4g48\") pod \"console-operator-85b454874b-rl7db\" (UID: \"d26dd5f9-339d-4123-b5e7-bf490e802699\") " pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.045898 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.045875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8p4\" (UniqueName: \"kubernetes.io/projected/e4f87f05-7a3b-4ca1-b9cd-49d10dabf520-kube-api-access-tg8p4\") pod \"service-ca-operator-859f94c4b-gwb7t\" (UID: \"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.086693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.086664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" Sep 09 15:55:39.097739 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.097711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" Sep 09 15:55:39.103571 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.103545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:39.274079 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.274000 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl"] Sep 09 15:55:39.276676 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:55:39.276653 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3aaeaa_985d_4ea4_a623_2522319a941a.slice/crio-7c18bf94aeb241c7166334ef69dea8e674fa37e88a494b24e1f7c8fccb138632 WatchSource:0}: Error finding container 7c18bf94aeb241c7166334ef69dea8e674fa37e88a494b24e1f7c8fccb138632: Status 404 returned error can't find the container with id 7c18bf94aeb241c7166334ef69dea8e674fa37e88a494b24e1f7c8fccb138632 Sep 09 15:55:39.362235 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.362139 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gt8ms"] Sep 09 15:55:39.364320 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:55:39.364298 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeafd31_c7ce_4911_98bd_773a62b87ba5.slice/crio-7139d049a5475a946a4bbbebd6fa9a33a3ebdb9f7a0f4b578c6188db5febb7f5 WatchSource:0}: Error finding container 7139d049a5475a946a4bbbebd6fa9a33a3ebdb9f7a0f4b578c6188db5febb7f5: Status 404 returned error can't find the container with id 7139d049a5475a946a4bbbebd6fa9a33a3ebdb9f7a0f4b578c6188db5febb7f5 Sep 09 15:55:39.385857 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.385836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t"] Sep 09 15:55:39.388176 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.388157 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-rl7db"] Sep 09 15:55:39.395354 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:55:39.395328 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f87f05_7a3b_4ca1_b9cd_49d10dabf520.slice/crio-65b7da8ed0a4f412e95a46ac7794fc617d60ebe632d89eb2ed66f998d616e366 WatchSource:0}: Error finding container 65b7da8ed0a4f412e95a46ac7794fc617d60ebe632d89eb2ed66f998d616e366: Status 404 returned error can't find the container with id 65b7da8ed0a4f412e95a46ac7794fc617d60ebe632d89eb2ed66f998d616e366 Sep 09 15:55:39.395659 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:55:39.395634 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26dd5f9_339d_4123_b5e7_bf490e802699.slice/crio-1c8d2f22d27676a88167dae6c5b2859ab602ac9cd3b5ea5033af8a710d314dd2 WatchSource:0}: Error finding container 1c8d2f22d27676a88167dae6c5b2859ab602ac9cd3b5ea5033af8a710d314dd2: Status 404 returned error can't find the container with id 1c8d2f22d27676a88167dae6c5b2859ab602ac9cd3b5ea5033af8a710d314dd2 Sep 09 15:55:39.439641 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.439615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:39.439761 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.439674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:39.439817 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:39.439765 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 15:55:39.439817 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:39.439793 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:39.439915 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:39.439826 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls podName:11c3ccaa-3771-4320-8342-1e4bf1c839c2 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:40.439807744 +0000 UTC m=+259.780999568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cx7tq" (UID: "11c3ccaa-3771-4320-8342-1e4bf1c839c2") : secret "samples-operator-tls" not found Sep 09 15:55:39.439915 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:39.439844 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:40.439835371 +0000 UTC m=+259.781027181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:39.867130 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.867056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" event={"ID":"7c3aaeaa-985d-4ea4-a623-2522319a941a","Type":"ContainerStarted","Data":"7c18bf94aeb241c7166334ef69dea8e674fa37e88a494b24e1f7c8fccb138632"} Sep 09 15:55:39.868423 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.868383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rl7db" event={"ID":"d26dd5f9-339d-4123-b5e7-bf490e802699","Type":"ContainerStarted","Data":"1c8d2f22d27676a88167dae6c5b2859ab602ac9cd3b5ea5033af8a710d314dd2"} Sep 09 15:55:39.869672 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.869640 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" event={"ID":"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520","Type":"ContainerStarted","Data":"65b7da8ed0a4f412e95a46ac7794fc617d60ebe632d89eb2ed66f998d616e366"} Sep 09 15:55:39.871226 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.871205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" event={"ID":"1eeafd31-c7ce-4911-98bd-773a62b87ba5","Type":"ContainerStarted","Data":"a07913b92afb051f2520476bf7fd9cf9ba4716236766812079352842562fdeee"} Sep 09 15:55:39.871341 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.871246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" event={"ID":"1eeafd31-c7ce-4911-98bd-773a62b87ba5","Type":"ContainerStarted","Data":"7139d049a5475a946a4bbbebd6fa9a33a3ebdb9f7a0f4b578c6188db5febb7f5"} Sep 09 15:55:39.889272 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:39.889213 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-gt8ms" podStartSLOduration=1.8891956479999998 podStartE2EDuration="1.889195648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:39.88910976 +0000 UTC m=+259.230301589" watchObservedRunningTime="2026-09-09 15:55:39.889195648 +0000 UTC m=+259.230387475" Sep 09 15:55:40.448453 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:40.448403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:40.448567 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:40.448539 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:40.448625 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:40.448582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:40.448625 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:40.448609 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:42.448588917 +0000 UTC m=+261.789780741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:40.448738 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:40.448678 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 15:55:40.448738 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:40.448727 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls podName:11c3ccaa-3771-4320-8342-1e4bf1c839c2 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:42.44871235 +0000 UTC m=+261.789904163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cx7tq" (UID: "11c3ccaa-3771-4320-8342-1e4bf1c839c2") : secret "samples-operator-tls" not found Sep 09 15:55:40.876702 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:40.876665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" event={"ID":"7c3aaeaa-985d-4ea4-a623-2522319a941a","Type":"ContainerStarted","Data":"b7bb4933dc9bf36a98f7ba541965218eae9abb13231f2f0200e9df065d9957fd"} Sep 09 15:55:40.892056 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:40.892002 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rjthl" podStartSLOduration=1.7969162239999998 podStartE2EDuration="2.891982062s" totalImagesPullingTime="1.095065838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:40.891899994 +0000 UTC m=+260.233091832" watchObservedRunningTime="2026-09-09 15:55:40.891982062 +0000 UTC m=+260.233173885" Sep 09 15:55:42.465641 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.465583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:42.466103 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.465693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:42.466103 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:42.465745 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:42.466103 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:42.465798 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 15:55:42.466103 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:42.465818 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.465802388 +0000 UTC m=+265.806994211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:42.466103 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:42.465860 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls podName:11c3ccaa-3771-4320-8342-1e4bf1c839c2 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.465843187 +0000 UTC m=+265.807035002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cx7tq" (UID: "11c3ccaa-3771-4320-8342-1e4bf1c839c2") : secret "samples-operator-tls" not found Sep 09 15:55:42.883487 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.883454 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/0.log" Sep 09 15:55:42.883652 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.883504 2562 generic.go:356] "Generic (PLEG): container finished" podID="d26dd5f9-339d-4123-b5e7-bf490e802699" containerID="cd7872283b8d0fac6df6181e457764558a6f8f17d7f87c7ae90d9803cb195d14" exitCode=255 Sep 09 15:55:42.883652 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.883576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rl7db" event={"ID":"d26dd5f9-339d-4123-b5e7-bf490e802699","Type":"ContainerDied","Data":"cd7872283b8d0fac6df6181e457764558a6f8f17d7f87c7ae90d9803cb195d14"} Sep 09 15:55:42.883803 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.883786 2562 scope.go:117] "RemoveContainer" containerID="cd7872283b8d0fac6df6181e457764558a6f8f17d7f87c7ae90d9803cb195d14" Sep 09 15:55:42.885143 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.885119 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" event={"ID":"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520","Type":"ContainerStarted","Data":"c405ac86ea2b9e8cb79341281484d04b4d5439f67cc57640f2e385b563beae32"} Sep 09 15:55:42.913599 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:42.913559 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" podStartSLOduration=2.213466031 podStartE2EDuration="4.91354694s" totalImagesPullingTime="2.700080909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:42.913251058 +0000 UTC m=+262.254442888" watchObservedRunningTime="2026-09-09 15:55:42.91354694 +0000 UTC m=+262.254738768" Sep 09 15:55:43.062237 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.062213 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck"] Sep 09 15:55:43.065010 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.064995 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" Sep 09 15:55:43.067730 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.067697 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 15:55:43.067839 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.067785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-c2x7j\"" Sep 09 15:55:43.067839 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.067790 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 15:55:43.070030 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.070010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mtx\" (UniqueName: \"kubernetes.io/projected/788fb6c6-44f6-469e-86e4-6bd076616049-kube-api-access-29mtx\") pod \"migrator-fdbb8fdff-48wck\" (UID: \"788fb6c6-44f6-469e-86e4-6bd076616049\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" Sep 09 15:55:43.074939 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.074916 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck"] Sep 09 15:55:43.170727 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.170653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mtx\" (UniqueName: \"kubernetes.io/projected/788fb6c6-44f6-469e-86e4-6bd076616049-kube-api-access-29mtx\") pod \"migrator-fdbb8fdff-48wck\" (UID: \"788fb6c6-44f6-469e-86e4-6bd076616049\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" Sep 09 15:55:43.173339 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.173316 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2bnh8"] Sep 09 15:55:43.176224 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.176207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.178531 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.178510 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 15:55:43.178751 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.178731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 15:55:43.179178 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.179157 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 15:55:43.179285 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.179227 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 15:55:43.179362 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.179345 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqq2h\"" Sep 09 15:55:43.181481 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.181460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mtx\" (UniqueName: \"kubernetes.io/projected/788fb6c6-44f6-469e-86e4-6bd076616049-kube-api-access-29mtx\") pod \"migrator-fdbb8fdff-48wck\" (UID: \"788fb6c6-44f6-469e-86e4-6bd076616049\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" Sep 09 15:55:43.190659 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.190630 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bnh8"] Sep 09 15:55:43.271485 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.271441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.271646 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.271496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39132dab-f7cc-40b4-8f79-d4dd024cc581-crio-socket\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.271646 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.271563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.271646 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.271619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39132dab-f7cc-40b4-8f79-d4dd024cc581-data-volume\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.271827 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.271680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fk2\" (UniqueName: \"kubernetes.io/projected/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-api-access-m4fk2\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.372633 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.372633 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39132dab-f7cc-40b4-8f79-d4dd024cc581-crio-socket\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.372805 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39132dab-f7cc-40b4-8f79-d4dd024cc581-crio-socket\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.372805 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.372723 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:43.372805 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.372915 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.372804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:43.872785054 +0000 UTC m=+263.213976877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:43.372915 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" Sep 09 15:55:43.372915 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.372829 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 15:55:43.372915 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.372881 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:43.872866899 +0000 UTC m=+263.214058720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : configmap "kube-rbac-proxy" not found Sep 09 15:55:43.372915 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39132dab-f7cc-40b4-8f79-d4dd024cc581-data-volume\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.373158 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.372918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fk2\" (UniqueName: \"kubernetes.io/projected/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-api-access-m4fk2\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.373253 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.373234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39132dab-f7cc-40b4-8f79-d4dd024cc581-data-volume\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.383127 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.383101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fk2\" (UniqueName: \"kubernetes.io/projected/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-api-access-m4fk2\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.637297 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.637271 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck"] Sep 09 15:55:43.639582 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:55:43.639546 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788fb6c6_44f6_469e_86e4_6bd076616049.slice/crio-0889d75ecf42c0c4d993e5a5accaaecb58ee0a284bf38e10fb829b04c5760b3f WatchSource:0}: Error finding container 0889d75ecf42c0c4d993e5a5accaaecb58ee0a284bf38e10fb829b04c5760b3f: Status 404 returned error can't find the container with id 0889d75ecf42c0c4d993e5a5accaaecb58ee0a284bf38e10fb829b04c5760b3f Sep 09 15:55:43.877134 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.877094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.877311 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.877156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:43.877311 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.877250 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 15:55:43.877311 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.877270 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:43.877311 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.877303 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:44.877288718 +0000 UTC m=+264.218480529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : configmap "kube-rbac-proxy" not found Sep 09 15:55:43.877531 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.877335 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:44.877316658 +0000 UTC m=+264.218508488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:43.890373 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.890322 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 15:55:43.890733 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.890719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/0.log" Sep 09 15:55:43.890784 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.890755 2562 generic.go:356] "Generic (PLEG): container finished" podID="d26dd5f9-339d-4123-b5e7-bf490e802699" containerID="889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5" exitCode=255 Sep 09 15:55:43.890816 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.890787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rl7db" event={"ID":"d26dd5f9-339d-4123-b5e7-bf490e802699","Type":"ContainerDied","Data":"889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5"} Sep 09 15:55:43.890888 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.890874 2562 scope.go:117] "RemoveContainer" containerID="cd7872283b8d0fac6df6181e457764558a6f8f17d7f87c7ae90d9803cb195d14" Sep 09 15:55:43.891071 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.891052 2562 scope.go:117] "RemoveContainer" containerID="889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5" Sep 09 15:55:43.891258 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:43.891238 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rl7db_openshift-console-operator(d26dd5f9-339d-4123-b5e7-bf490e802699)\"" pod="openshift-console-operator/console-operator-85b454874b-rl7db" podUID="d26dd5f9-339d-4123-b5e7-bf490e802699" Sep 09 15:55:43.892023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:43.892006 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" event={"ID":"788fb6c6-44f6-469e-86e4-6bd076616049","Type":"ContainerStarted","Data":"0889d75ecf42c0c4d993e5a5accaaecb58ee0a284bf38e10fb829b04c5760b3f"} Sep 09 15:55:44.885946 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:44.885907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:44.886319 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:44.885963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:44.886319 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:44.886025 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 15:55:44.886319 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:44.886043 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:44.886319 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:44.886078 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.8860649 +0000 UTC m=+266.227256711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : configmap "kube-rbac-proxy" not found Sep 09 15:55:44.886319 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:44.886096 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:46.886083107 +0000 UTC m=+266.227274938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:44.896176 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:44.896156 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 15:55:44.896553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:44.896534 2562 scope.go:117] "RemoveContainer" containerID="889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5" Sep 09 15:55:44.896743 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:44.896724 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rl7db_openshift-console-operator(d26dd5f9-339d-4123-b5e7-bf490e802699)\"" pod="openshift-console-operator/console-operator-85b454874b-rl7db" podUID="d26dd5f9-339d-4123-b5e7-bf490e802699" Sep 09 15:55:46.142770 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.142744 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nt5bs_b783ea01-682c-4b85-adea-0c3cff2e3734/dns-node-resolver/0.log" Sep 09 15:55:46.501023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.500927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:46.501023 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.500993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:46.501212 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.501084 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 15:55:46.501212 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.501127 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:46.501212 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.501150 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls podName:11c3ccaa-3771-4320-8342-1e4bf1c839c2 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:54.501134656 +0000 UTC m=+273.842326482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cx7tq" (UID: "11c3ccaa-3771-4320-8342-1e4bf1c839c2") : secret "samples-operator-tls" not found Sep 09 15:55:46.501212 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.501166 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:54.501155509 +0000 UTC m=+273.842347334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:46.903527 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.903499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:46.903663 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.903646 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 15:55:46.903724 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.903657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:46.903724 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.903713 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:50.903693133 +0000 UTC m=+270.244884963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : configmap "kube-rbac-proxy" not found Sep 09 15:55:46.903847 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.903808 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:46.903901 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:46.903874 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:50.903856033 +0000 UTC m=+270.245047912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:46.904875 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:46.904754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" event={"ID":"788fb6c6-44f6-469e-86e4-6bd076616049","Type":"ContainerStarted","Data":"7f4bdd00aa5f0df1987934ffa190b1ba7146e4fe30edf32007e59288f5182b41"} Sep 09 15:55:47.142456 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:47.142404 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6m4g_65c46cff-df1c-4afe-9ee8-a8e48413f0c1/node-ca/0.log" Sep 09 15:55:47.909396 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:47.909355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" event={"ID":"788fb6c6-44f6-469e-86e4-6bd076616049","Type":"ContainerStarted","Data":"ea6862984bc3b3b3a618e7e4101cb9929cb7595b1f71db8d8ff5b9d2bdb16b70"} Sep 09 15:55:47.932339 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:47.932288 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-48wck" podStartSLOduration=1.742208242 podStartE2EDuration="4.932273663s" totalImagesPullingTime="3.190065421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:47.930845885 +0000 UTC m=+267.272037726" watchObservedRunningTime="2026-09-09 15:55:47.932273663 +0000 UTC m=+267.273465490" Sep 09 15:55:47.942772 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:47.942746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srs75_9bcb88f1-3276-4896-bbe6-e6b29f66d4a1/serve-healthcheck-canary/0.log" Sep 09 15:55:48.142387 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:48.142356 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-48wck_788fb6c6-44f6-469e-86e4-6bd076616049/migrator/0.log" Sep 09 15:55:48.342509 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:48.342437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-48wck_788fb6c6-44f6-469e-86e4-6bd076616049/graceful-termination/0.log" Sep 09 15:55:49.104294 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:49.104262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:49.104703 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:49.104311 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:55:49.104703 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:49.104667 2562 scope.go:117] "RemoveContainer" containerID="889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5" Sep 09 15:55:49.104837 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:49.104817 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-rl7db_openshift-console-operator(d26dd5f9-339d-4123-b5e7-bf490e802699)\"" pod="openshift-console-operator/console-operator-85b454874b-rl7db" podUID="d26dd5f9-339d-4123-b5e7-bf490e802699" Sep 09 15:55:50.937914 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:50.937878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:50.938292 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:50.937934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:50.938292 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:50.938027 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 15:55:50.938292 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:50.938027 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:50.938292 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:50.938076 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:58.938062281 +0000 UTC m=+278.279254088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : configmap "kube-rbac-proxy" not found Sep 09 15:55:50.938292 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:50.938089 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:55:58.938082811 +0000 UTC m=+278.279274618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:54.569013 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.568977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:54.569465 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.569034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:55:54.569465 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:54.569311 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:54.569465 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:54.569368 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls podName:3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7 nodeName:}" failed. No retries permitted until 2026-09-09 15:56:10.569353855 +0000 UTC m=+289.910545678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-k6ppp" (UID: "3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7") : secret "cluster-monitoring-operator-tls" not found Sep 09 15:55:54.571608 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.571568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c3ccaa-3771-4320-8342-1e4bf1c839c2-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cx7tq\" (UID: \"11c3ccaa-3771-4320-8342-1e4bf1c839c2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:54.614133 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.614090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" Sep 09 15:55:54.881013 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.880932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq"] Sep 09 15:55:54.930626 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:54.930595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" event={"ID":"11c3ccaa-3771-4320-8342-1e4bf1c839c2","Type":"ContainerStarted","Data":"8990cbab0c7dba13e3ec8084e13f3b189b89165a5d3b8d868fe8305493d580bb"} Sep 09 15:55:56.937876 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:56.937796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" event={"ID":"11c3ccaa-3771-4320-8342-1e4bf1c839c2","Type":"ContainerStarted","Data":"f19395fd828f7c16a25be99d986a4aeea23cb3eda9766a751230740b0ac4d530"} Sep 09 15:55:56.937876 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:56.937858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" event={"ID":"11c3ccaa-3771-4320-8342-1e4bf1c839c2","Type":"ContainerStarted","Data":"76d06230137eb268f467dd21b16054ef5925021c5b102f285e3a3cda8027772d"} Sep 09 15:55:56.956580 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:56.956512 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cx7tq" podStartSLOduration=17.266601311 podStartE2EDuration="18.956491248s" totalImagesPullingTime="1.689889937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:56.956356456 +0000 UTC m=+276.297548285" watchObservedRunningTime="2026-09-09 15:55:56.956491248 +0000 UTC m=+276.297683075" Sep 09 15:55:59.003263 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:59.003228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:59.003667 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:59.003316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:55:59.003667 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:59.003397 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 15:55:59.003667 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:55:59.003468 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls podName:39132dab-f7cc-40b4-8f79-d4dd024cc581 nodeName:}" failed. No retries permitted until 2026-09-09 15:56:15.003455039 +0000 UTC m=+294.344646845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2bnh8" (UID: "39132dab-f7cc-40b4-8f79-d4dd024cc581") : secret "insights-runtime-extractor-tls" not found Sep 09 15:55:59.003822 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:55:59.003802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39132dab-f7cc-40b4-8f79-d4dd024cc581-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:56:01.184510 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:01.184470 2562 scope.go:117] "RemoveContainer" containerID="889fa68c1ef66d95f5459f5f07c89d88d7e2dde44e31ce8cf94541a31e5130a5" Sep 09 15:56:01.958338 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:01.958310 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 15:56:01.958525 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:01.958356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-rl7db" event={"ID":"d26dd5f9-339d-4123-b5e7-bf490e802699","Type":"ContainerStarted","Data":"266207f08350fe55a8aae2ff53cf88ab8284bdfe0c05e9f96f8ba7bb0b6a7f37"} Sep 09 15:56:01.958689 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:01.958637 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:56:01.975991 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:01.975933 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-rl7db" podStartSLOduration=21.2785787 podStartE2EDuration="23.97591358s" totalImagesPullingTime="2.69733488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:01.974502154 +0000 UTC m=+281.315693982" watchObservedRunningTime="2026-09-09 15:56:01.97591358 +0000 UTC m=+281.317105409" Sep 09 15:56:02.287943 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:02.287870 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-rl7db" Sep 09 15:56:04.509340 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.509311 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-rmfsv"] Sep 09 15:56:04.512347 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.512324 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:04.514796 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.514774 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 15:56:04.514898 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.514781 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xvmq4\"" Sep 09 15:56:04.514978 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.514902 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 15:56:04.523437 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.523398 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rmfsv"] Sep 09 15:56:04.639353 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:04.639317 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" Sep 09 15:56:04.650514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.650483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkcl\" (UniqueName: \"kubernetes.io/projected/e326c70e-e584-422a-b300-a8bcabd58ebb-kube-api-access-nbkcl\") pod \"downloads-85d98c998f-rmfsv\" (UID: \"e326c70e-e584-422a-b300-a8bcabd58ebb\") " pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:04.751196 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.751157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkcl\" (UniqueName: \"kubernetes.io/projected/e326c70e-e584-422a-b300-a8bcabd58ebb-kube-api-access-nbkcl\") pod \"downloads-85d98c998f-rmfsv\" (UID: \"e326c70e-e584-422a-b300-a8bcabd58ebb\") " pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:04.759735 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.759684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkcl\" (UniqueName: \"kubernetes.io/projected/e326c70e-e584-422a-b300-a8bcabd58ebb-kube-api-access-nbkcl\") pod \"downloads-85d98c998f-rmfsv\" (UID: \"e326c70e-e584-422a-b300-a8bcabd58ebb\") " pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:04.821400 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.821369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:04.967924 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:04.967897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:05.091224 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:05.091202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rmfsv"] Sep 09 15:56:05.093561 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:05.093537 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode326c70e_e584_422a_b300_a8bcabd58ebb.slice/crio-d662f5539bbd9c4c563ffe7ff3711fca7340e85df6aada1490253c98cba4aff0 WatchSource:0}: Error finding container d662f5539bbd9c4c563ffe7ff3711fca7340e85df6aada1490253c98cba4aff0: Status 404 returned error can't find the container with id d662f5539bbd9c4c563ffe7ff3711fca7340e85df6aada1490253c98cba4aff0 Sep 09 15:56:05.971731 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:05.971694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rmfsv" event={"ID":"e326c70e-e584-422a-b300-a8bcabd58ebb","Type":"ContainerStarted","Data":"d662f5539bbd9c4c563ffe7ff3711fca7340e85df6aada1490253c98cba4aff0"} Sep 09 15:56:07.879900 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.879865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:07.882273 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.882248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"image-registry-5494898f59-lqwlc\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:07.971113 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.971076 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wwhjq\"" Sep 09 15:56:07.979257 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.979230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:07.980728 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.980699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:56:07.980865 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.980751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:56:07.983352 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.983308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef9d60a4-51da-4d45-b1fa-64a1d251965f-metrics-tls\") pod \"dns-default-wbgbv\" (UID: \"ef9d60a4-51da-4d45-b1fa-64a1d251965f\") " pod="openshift-dns/dns-default-wbgbv" Sep 09 15:56:07.983514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.983444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c40bf4af-9876-408d-9527-5115d436f397-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-d7kfz\" (UID: \"c40bf4af-9876-408d-9527-5115d436f397\") " pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:56:07.986477 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.986458 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mzxcf\"" Sep 09 15:56:07.986575 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.986481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5xfw9\"" Sep 09 15:56:07.994587 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.994559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" Sep 09 15:56:07.994682 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:07.994599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbgbv" Sep 09 15:56:08.257157 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.257032 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:56:08.261617 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:08.261585 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f9c5e2_bb0b_498d_82db_a203c5a1feea.slice/crio-8a163d939e17536ba60bc87bbc22a09197a21f5310c4ba0c22ca4213b0ea2d10 WatchSource:0}: Error finding container 8a163d939e17536ba60bc87bbc22a09197a21f5310c4ba0c22ca4213b0ea2d10: Status 404 returned error can't find the container with id 8a163d939e17536ba60bc87bbc22a09197a21f5310c4ba0c22ca4213b0ea2d10 Sep 09 15:56:08.282694 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.282670 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-d7kfz"] Sep 09 15:56:08.285714 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:08.285684 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40bf4af_9876_408d_9527_5115d436f397.slice/crio-f101d20edf65210046fcb46151c59374189dcb1d851783f64c4c55b4a502761d WatchSource:0}: Error finding container f101d20edf65210046fcb46151c59374189dcb1d851783f64c4c55b4a502761d: Status 404 returned error can't find the container with id f101d20edf65210046fcb46151c59374189dcb1d851783f64c4c55b4a502761d Sep 09 15:56:08.286091 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.286019 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbgbv"] Sep 09 15:56:08.292546 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:08.292515 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9d60a4_51da_4d45_b1fa_64a1d251965f.slice/crio-035f52ff3dad172863b770f424417fbe67681d498c4a5d660d1da42ad217007c WatchSource:0}: Error finding container 035f52ff3dad172863b770f424417fbe67681d498c4a5d660d1da42ad217007c: Status 404 returned error can't find the container with id 035f52ff3dad172863b770f424417fbe67681d498c4a5d660d1da42ad217007c Sep 09 15:56:08.987216 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.987177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbgbv" event={"ID":"ef9d60a4-51da-4d45-b1fa-64a1d251965f","Type":"ContainerStarted","Data":"035f52ff3dad172863b770f424417fbe67681d498c4a5d660d1da42ad217007c"} Sep 09 15:56:08.988640 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.988609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" event={"ID":"c40bf4af-9876-408d-9527-5115d436f397","Type":"ContainerStarted","Data":"f101d20edf65210046fcb46151c59374189dcb1d851783f64c4c55b4a502761d"} Sep 09 15:56:08.990245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.990217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" event={"ID":"79f9c5e2-bb0b-498d-82db-a203c5a1feea","Type":"ContainerStarted","Data":"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5"} Sep 09 15:56:08.990371 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.990261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" event={"ID":"79f9c5e2-bb0b-498d-82db-a203c5a1feea","Type":"ContainerStarted","Data":"8a163d939e17536ba60bc87bbc22a09197a21f5310c4ba0c22ca4213b0ea2d10"} Sep 09 15:56:08.990371 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:08.990362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:09.010083 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:09.010036 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" podStartSLOduration=288.010024181 podStartE2EDuration="4m48.010024181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:09.008722938 +0000 UTC m=+288.349914768" watchObservedRunningTime="2026-09-09 15:56:09.010024181 +0000 UTC m=+288.351216009" Sep 09 15:56:10.604474 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:10.604431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:56:10.607172 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:10.607147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-k6ppp\" (UID: \"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:56:10.802727 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:10.802689 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" Sep 09 15:56:11.082502 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:11.082467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp"] Sep 09 15:56:11.085844 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:11.085815 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3677b106_49bd_4ac2_8f4e_0e7d2bd4dea7.slice/crio-8aadb730d94c4fcdcccbd08a289dc6e7887ae5386148ce3d41cb5b890ee986d7 WatchSource:0}: Error finding container 8aadb730d94c4fcdcccbd08a289dc6e7887ae5386148ce3d41cb5b890ee986d7: Status 404 returned error can't find the container with id 8aadb730d94c4fcdcccbd08a289dc6e7887ae5386148ce3d41cb5b890ee986d7 Sep 09 15:56:12.004159 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.004114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" event={"ID":"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7","Type":"ContainerStarted","Data":"8aadb730d94c4fcdcccbd08a289dc6e7887ae5386148ce3d41cb5b890ee986d7"} Sep 09 15:56:12.203693 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.203657 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:12.222945 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.222918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:12.223102 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.223066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.225941 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.225716 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 15:56:12.225941 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.225794 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 15:56:12.226911 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.226886 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 15:56:12.226911 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.226903 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 15:56:12.227118 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.227050 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 15:56:12.227118 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.227063 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2p968\"" Sep 09 15:56:12.319875 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.319790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.319875 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.319841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.320081 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.319905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.320081 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.320017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpntk\" (UniqueName: \"kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.320081 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.320056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.320243 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.320083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421176 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpntk\" (UniqueName: \"kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421368 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421368 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421368 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421322 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421368 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.421608 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.421383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.422220 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.422195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.422317 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.422234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.422317 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.422249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.425287 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.424828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.425287 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.424973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.429285 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.429261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpntk\" (UniqueName: \"kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk\") pod \"console-8c9fb5668-ssjkr\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.540450 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.540402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:12.825068 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:12.825035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:12.828182 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:12.828156 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ac3678_1c2e_4578_b39d_0645b23677ef.slice/crio-870f8eec50ec645ed154fb5c0c4e014cb5462350e7bbca0e39856ca085b65c71 WatchSource:0}: Error finding container 870f8eec50ec645ed154fb5c0c4e014cb5462350e7bbca0e39856ca085b65c71: Status 404 returned error can't find the container with id 870f8eec50ec645ed154fb5c0c4e014cb5462350e7bbca0e39856ca085b65c71 Sep 09 15:56:13.009535 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:13.009432 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c9fb5668-ssjkr" event={"ID":"31ac3678-1c2e-4578-b39d-0645b23677ef","Type":"ContainerStarted","Data":"870f8eec50ec645ed154fb5c0c4e014cb5462350e7bbca0e39856ca085b65c71"} Sep 09 15:56:14.016972 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.016891 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbgbv" event={"ID":"ef9d60a4-51da-4d45-b1fa-64a1d251965f","Type":"ContainerStarted","Data":"d8b57b80ad341b08e22342bf1ace1b6131607af3832f27660cc80942c045a697"} Sep 09 15:56:14.016972 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.016959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbgbv" event={"ID":"ef9d60a4-51da-4d45-b1fa-64a1d251965f","Type":"ContainerStarted","Data":"3d85ea5878381072302ed1a2be9e51b17ab7f3fce06b448eec38663b8ee001dc"} Sep 09 15:56:14.017454 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.017286 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wbgbv" Sep 09 15:56:14.019227 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.019201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" event={"ID":"c40bf4af-9876-408d-9527-5115d436f397","Type":"ContainerStarted","Data":"5e5deba86024dadda4bc7c2374a29341604199dd2b80daad4bbd5f58690cf2bf"} Sep 09 15:56:14.034706 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.034653 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wbgbv" podStartSLOduration=251.557195751 podStartE2EDuration="4m17.034636516s" totalImagesPullingTime="5.477440765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:14.03448672 +0000 UTC m=+293.375678547" watchObservedRunningTime="2026-09-09 15:56:14.034636516 +0000 UTC m=+293.375828345" Sep 09 15:56:14.053132 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:14.053085 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-d7kfz" podStartSLOduration=277.574211784 podStartE2EDuration="4m43.053068608s" totalImagesPullingTime="5.478856824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:14.052436833 +0000 UTC m=+293.393628657" watchObservedRunningTime="2026-09-09 15:56:14.053068608 +0000 UTC m=+293.394260439" Sep 09 15:56:15.043252 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:15.043127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:56:15.047058 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:15.047032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39132dab-f7cc-40b4-8f79-d4dd024cc581-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2bnh8\" (UID: \"39132dab-f7cc-40b4-8f79-d4dd024cc581\") " pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:56:15.291999 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:15.291961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2bnh8" Sep 09 15:56:19.183836 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:19.183790 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wpwhx" podUID="337bfd9d-63ad-43b9-9b6f-e79aab96a48f" Sep 09 15:56:23.565463 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:23.565361 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 15:56:23.566239 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:23.566153 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 15:56:23.585684 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:23.585663 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 15:56:23.825662 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:23.825527 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2bnh8"] Sep 09 15:56:23.830157 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:23.830123 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39132dab_f7cc_40b4_8f79_d4dd024cc581.slice/crio-e39ccfd83eaf91704ca40c7adafad678ff90503de04c0ec9ca2f17fccfd04c45 WatchSource:0}: Error finding container e39ccfd83eaf91704ca40c7adafad678ff90503de04c0ec9ca2f17fccfd04c45: Status 404 returned error can't find the container with id e39ccfd83eaf91704ca40c7adafad678ff90503de04c0ec9ca2f17fccfd04c45 Sep 09 15:56:23.903380 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:23.903329 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 15:56:24.026373 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.025562 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wbgbv" Sep 09 15:56:24.050362 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.050329 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf"] Sep 09 15:56:24.060471 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.060440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bnh8" event={"ID":"39132dab-f7cc-40b4-8f79-d4dd024cc581","Type":"ContainerStarted","Data":"b4a310315b7c3db63a731bf6ddcb63723565a1da63b98a1b8484f7aa3f539dda"} Sep 09 15:56:24.060640 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.060488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bnh8" event={"ID":"39132dab-f7cc-40b4-8f79-d4dd024cc581","Type":"ContainerStarted","Data":"e39ccfd83eaf91704ca40c7adafad678ff90503de04c0ec9ca2f17fccfd04c45"} Sep 09 15:56:24.060640 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.060593 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:24.064183 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.064154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf"] Sep 09 15:56:24.064994 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.064485 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j226j\"" Sep 09 15:56:24.064994 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.064938 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 15:56:24.067270 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.066092 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rmfsv" event={"ID":"e326c70e-e584-422a-b300-a8bcabd58ebb","Type":"ContainerStarted","Data":"0356c464656d9f681b23c1cb843432883a6c93499e723525328820fde78e015e"} Sep 09 15:56:24.067270 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.067056 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:24.070968 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.070243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" event={"ID":"3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7","Type":"ContainerStarted","Data":"f92cdb0cb46628f2ae498cdfc7c66f935bbc50d47cb1cfdd5cf174d029253fb7"} Sep 09 15:56:24.070968 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.070926 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-rmfsv" Sep 09 15:56:24.076621 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.076097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c9fb5668-ssjkr" event={"ID":"31ac3678-1c2e-4578-b39d-0645b23677ef","Type":"ContainerStarted","Data":"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0"} Sep 09 15:56:24.120352 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.120305 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-k6ppp" podStartSLOduration=33.658786034 podStartE2EDuration="46.120283876s" totalImagesPullingTime="12.461497842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:24.100638898 +0000 UTC m=+303.441830729" watchObservedRunningTime="2026-09-09 15:56:24.120283876 +0000 UTC m=+303.461475704" Sep 09 15:56:24.120996 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.120949 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c9fb5668-ssjkr" podStartSLOduration=1.397560908 podStartE2EDuration="12.120933406s" totalImagesPullingTime="10.723372498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:24.119304958 +0000 UTC m=+303.460496800" watchObservedRunningTime="2026-09-09 15:56:24.120933406 +0000 UTC m=+303.462125238" Sep 09 15:56:24.133995 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.133948 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-rmfsv" podStartSLOduration=1.613664097 podStartE2EDuration="20.133928672s" totalImagesPullingTime="18.520264575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:24.132863104 +0000 UTC m=+303.474054935" watchObservedRunningTime="2026-09-09 15:56:24.133928672 +0000 UTC m=+303.475120501" Sep 09 15:56:24.219140 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.219105 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4ed807b-b4ef-445d-82b3-01003ac5a1bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-lncpf\" (UID: \"d4ed807b-b4ef-445d-82b3-01003ac5a1bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:24.320610 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.320508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4ed807b-b4ef-445d-82b3-01003ac5a1bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-lncpf\" (UID: \"d4ed807b-b4ef-445d-82b3-01003ac5a1bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:24.324022 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.323962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d4ed807b-b4ef-445d-82b3-01003ac5a1bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-lncpf\" (UID: \"d4ed807b-b4ef-445d-82b3-01003ac5a1bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:24.387507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.387464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:24.761237 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:24.761109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf"] Sep 09 15:56:24.765377 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:24.765341 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ed807b_b4ef_445d_82b3_01003ac5a1bb.slice/crio-15053c66187395320a85c9d0bc1d3852bcfc80961d7f0da6433accf1c472e3a0 WatchSource:0}: Error finding container 15053c66187395320a85c9d0bc1d3852bcfc80961d7f0da6433accf1c472e3a0: Status 404 returned error can't find the container with id 15053c66187395320a85c9d0bc1d3852bcfc80961d7f0da6433accf1c472e3a0 Sep 09 15:56:25.084923 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:25.084738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bnh8" event={"ID":"39132dab-f7cc-40b4-8f79-d4dd024cc581","Type":"ContainerStarted","Data":"c996d4fc10b658a13d6b408da5350f0d6556090b0149796e4d37920e03ee6005"} Sep 09 15:56:25.093265 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:25.093202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" event={"ID":"d4ed807b-b4ef-445d-82b3-01003ac5a1bb","Type":"ContainerStarted","Data":"15053c66187395320a85c9d0bc1d3852bcfc80961d7f0da6433accf1c472e3a0"} Sep 09 15:56:27.217091 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:27.217055 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:56:27.222373 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:27.222334 2562 patch_prober.go:28] interesting pod/image-registry-5494898f59-lqwlc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 15:56:27.222547 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:27.222396 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 15:56:28.108062 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.107986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" event={"ID":"d4ed807b-b4ef-445d-82b3-01003ac5a1bb","Type":"ContainerStarted","Data":"e9b9b1b53747ab016da5ec413836b3b28e2e47e83b20953d923d3e03598acc03"} Sep 09 15:56:28.108252 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.108194 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:28.110650 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.110601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2bnh8" event={"ID":"39132dab-f7cc-40b4-8f79-d4dd024cc581","Type":"ContainerStarted","Data":"6bef73e3923565f491f81b695bb237fa3a470a2d6a897de99d5e0198dc1e4922"} Sep 09 15:56:28.114072 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.114049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" Sep 09 15:56:28.123995 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.123949 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-lncpf" podStartSLOduration=1.868259441 podStartE2EDuration="4.123930495s" totalImagesPullingTime="2.255671054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:28.122522246 +0000 UTC m=+307.463714085" watchObservedRunningTime="2026-09-09 15:56:28.123930495 +0000 UTC m=+307.465122320" Sep 09 15:56:28.140280 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:28.140238 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2bnh8" podStartSLOduration=42.121636124 podStartE2EDuration="45.140220018s" totalImagesPullingTime="3.018583894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:28.138907882 +0000 UTC m=+307.480099710" watchObservedRunningTime="2026-09-09 15:56:28.140220018 +0000 UTC m=+307.481411880" Sep 09 15:56:30.183916 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:30.183879 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:56:32.321423 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:32.321377 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:32.541117 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:32.541058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:33.477498 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.477467 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2qk4f"] Sep 09 15:56:33.519237 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.519207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.521847 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.521790 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 15:56:33.521976 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.521894 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 15:56:33.521976 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.521911 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 15:56:33.522090 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.522042 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 15:56:33.523024 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.523003 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pqjbj\"" Sep 09 15:56:33.604850 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.604803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605018 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.604876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-textfile\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605018 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.604925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-sys\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605018 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.604978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605018 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.605005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-wtmp\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605154 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.605029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605154 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.605072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcd2\" (UniqueName: \"kubernetes.io/projected/66001ff9-94de-4765-88f4-450c64c01efc-kube-api-access-xhcd2\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605154 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.605101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-metrics-client-ca\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.605154 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.605126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-root\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.705809 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-wtmp\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.705809 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcd2\" (UniqueName: \"kubernetes.io/projected/66001ff9-94de-4765-88f4-450c64c01efc-kube-api-access-xhcd2\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-metrics-client-ca\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-root\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705972 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-wtmp\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-root\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706009 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.705989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-textfile\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-sys\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706179 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66001ff9-94de-4765-88f4-450c64c01efc-sys\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706448 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:33.706260 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 15:56:33.706448 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:33.706317 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls podName:66001ff9-94de-4765-88f4-450c64c01efc nodeName:}" failed. No retries permitted until 2026-09-09 15:56:34.20629795 +0000 UTC m=+313.547489776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls") pod "node-exporter-2qk4f" (UID: "66001ff9-94de-4765-88f4-450c64c01efc") : secret "node-exporter-tls" not found Sep 09 15:56:33.706448 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-textfile\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706668 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.706668 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.706634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66001ff9-94de-4765-88f4-450c64c01efc-metrics-client-ca\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.708237 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.708216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:33.714369 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:33.714342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcd2\" (UniqueName: \"kubernetes.io/projected/66001ff9-94de-4765-88f4-450c64c01efc-kube-api-access-xhcd2\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:34.210652 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.210616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:34.212978 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.212955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66001ff9-94de-4765-88f4-450c64c01efc-node-exporter-tls\") pod \"node-exporter-2qk4f\" (UID: \"66001ff9-94de-4765-88f4-450c64c01efc\") " pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:34.429069 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.429037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qk4f" Sep 09 15:56:34.471281 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:34.471252 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66001ff9_94de_4765_88f4_450c64c01efc.slice/crio-050244690e74eba1dd193bd1f53bb9385ade1184a9603411e9116c6fce07ef1f WatchSource:0}: Error finding container 050244690e74eba1dd193bd1f53bb9385ade1184a9603411e9116c6fce07ef1f: Status 404 returned error can't find the container with id 050244690e74eba1dd193bd1f53bb9385ade1184a9603411e9116c6fce07ef1f Sep 09 15:56:34.545283 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.545246 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 15:56:34.581132 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.581100 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 15:56:34.581294 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.581217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.583823 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.583801 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 15:56:34.584064 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.584048 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 15:56:34.584378 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.584351 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 15:56:34.584473 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.584457 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 15:56:34.584530 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.584506 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 15:56:34.584822 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.584806 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 15:56:34.585050 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.585035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 15:56:34.585291 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.585275 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 15:56:34.585752 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.585728 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 15:56:34.585864 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.585792 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27zgb\"" Sep 09 15:56:34.615027 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615027 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbfn\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-kube-api-access-lpbfn\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615156 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-web-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615386 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615386 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615386 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-out\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.615533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.615514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716450 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbfn\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-kube-api-access-lpbfn\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716754 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716754 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-web-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716754 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.716754 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.716625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.717139 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.717115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.717700 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.717678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.717887 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.717868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-out\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.718025 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.718006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.718151 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.718137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.718255 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.718241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719433 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.718678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719433 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719433 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9069f88-8c0c-46bf-8d0c-cfa299f54513-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719644 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719644 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.719972 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-web-config\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.720073 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.719975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.720773 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.720733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.720954 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.720930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.721117 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.721095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9069f88-8c0c-46bf-8d0c-cfa299f54513-config-out\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.721858 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.721829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.722068 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.722047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9069f88-8c0c-46bf-8d0c-cfa299f54513-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.723692 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.723671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbfn\" (UniqueName: \"kubernetes.io/projected/a9069f88-8c0c-46bf-8d0c-cfa299f54513-kube-api-access-lpbfn\") pod \"alertmanager-main-0\" (UID: \"a9069f88-8c0c-46bf-8d0c-cfa299f54513\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:34.890809 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:34.890769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 15:56:35.141929 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:35.141885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qk4f" event={"ID":"66001ff9-94de-4765-88f4-450c64c01efc","Type":"ContainerStarted","Data":"050244690e74eba1dd193bd1f53bb9385ade1184a9603411e9116c6fce07ef1f"} Sep 09 15:56:35.189925 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:35.189896 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 15:56:35.190819 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:35.190795 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9069f88_8c0c_46bf_8d0c_cfa299f54513.slice/crio-8ef4ddf69c4f633965baf915af718ccc02880a61e35ad98719be6bbadeb5cd25 WatchSource:0}: Error finding container 8ef4ddf69c4f633965baf915af718ccc02880a61e35ad98719be6bbadeb5cd25: Status 404 returned error can't find the container with id 8ef4ddf69c4f633965baf915af718ccc02880a61e35ad98719be6bbadeb5cd25 Sep 09 15:56:36.147010 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.146960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"8ef4ddf69c4f633965baf915af718ccc02880a61e35ad98719be6bbadeb5cd25"} Sep 09 15:56:36.431252 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.431180 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75b97b9786-k62bs"] Sep 09 15:56:36.450381 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.450336 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75b97b9786-k62bs"] Sep 09 15:56:36.450579 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.450481 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.453080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.452978 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 15:56:36.453080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.453037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 15:56:36.453080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.453045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 15:56:36.453080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.453043 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 15:56:36.453338 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.453046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kjfx8\"" Sep 09 15:56:36.453338 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.452985 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6fvq9ac664jcd\"" Sep 09 15:56:36.453338 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.452986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 15:56:36.542158 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542306 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215b1af2-8686-45db-a869-efb1ab5009b3-metrics-client-ca\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542306 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6lh\" (UniqueName: \"kubernetes.io/projected/215b1af2-8686-45db-a869-efb1ab5009b3-kube-api-access-fh6lh\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542306 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-grpc-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542306 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542467 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542467 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.542467 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.542346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643172 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215b1af2-8686-45db-a869-efb1ab5009b3-metrics-client-ca\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643324 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6lh\" (UniqueName: \"kubernetes.io/projected/215b1af2-8686-45db-a869-efb1ab5009b3-kube-api-access-fh6lh\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.643633 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.643331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-grpc-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.644082 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.644043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215b1af2-8686-45db-a869-efb1ab5009b3-metrics-client-ca\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.645982 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.645952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.646127 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.646106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.646169 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.646106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.646169 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.646154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.646475 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.646458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.646517 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.646477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/215b1af2-8686-45db-a869-efb1ab5009b3-secret-grpc-tls\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.650801 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.650783 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6lh\" (UniqueName: \"kubernetes.io/projected/215b1af2-8686-45db-a869-efb1ab5009b3-kube-api-access-fh6lh\") pod \"thanos-querier-75b97b9786-k62bs\" (UID: \"215b1af2-8686-45db-a869-efb1ab5009b3\") " pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:36.760656 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:36.760578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:37.050517 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:37.050491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75b97b9786-k62bs"] Sep 09 15:56:37.051516 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:37.051489 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215b1af2_8686_45db_a869_efb1ab5009b3.slice/crio-a403f80c34c2fa4fa12f7e97cfcc426dada68a39d16be0286e785a7d1be5cb4e WatchSource:0}: Error finding container a403f80c34c2fa4fa12f7e97cfcc426dada68a39d16be0286e785a7d1be5cb4e: Status 404 returned error can't find the container with id a403f80c34c2fa4fa12f7e97cfcc426dada68a39d16be0286e785a7d1be5cb4e Sep 09 15:56:37.152206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:37.152174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"a403f80c34c2fa4fa12f7e97cfcc426dada68a39d16be0286e785a7d1be5cb4e"} Sep 09 15:56:37.153683 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:37.153659 2562 generic.go:356] "Generic (PLEG): container finished" podID="a9069f88-8c0c-46bf-8d0c-cfa299f54513" containerID="20a86caf828c350b18a462f1b7f58a1345f6c1ebd620dcbf119b2074b2017507" exitCode=0 Sep 09 15:56:37.153798 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:37.153692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerDied","Data":"20a86caf828c350b18a462f1b7f58a1345f6c1ebd620dcbf119b2074b2017507"} Sep 09 15:56:37.221807 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:37.221785 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:39.696101 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.696062 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 15:56:39.716370 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.716197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.717595 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.717557 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 15:56:39.720655 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.720613 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 15:56:39.720791 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.720652 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 15:56:39.720791 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.720716 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 15:56:39.720916 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.720824 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 15:56:39.720916 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.720842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 15:56:39.721129 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721103 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9temgqbl2lugk\"" Sep 09 15:56:39.721255 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721227 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 15:56:39.721255 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721244 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 15:56:39.721372 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721268 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 15:56:39.721372 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721363 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 15:56:39.721514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721438 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 15:56:39.721514 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.721453 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zv8pg\"" Sep 09 15:56:39.722278 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.722222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 15:56:39.722760 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.722726 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 15:56:39.722855 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.722806 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 15:56:39.722986 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.722952 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 15:56:39.726606 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.726576 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 15:56:39.779438 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2c6m\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-kube-api-access-l2c6m\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779628 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.779935 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-config-out\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.779936 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-web-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.780017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.780045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.780087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.780117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.780204 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.780147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881212 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881393 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881393 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2c6m\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-kube-api-access-l2c6m\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881553 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.881797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882054 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.881959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882110 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-config-out\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882110 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-web-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882206 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882308 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882308 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.882308 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.882296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.883549 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.883455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.884726 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.883720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.884726 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.884546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.885543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.885739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.885813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.885814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.885908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.886041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.886267 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.886232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.887050 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.887028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27333cea-cd69-46ed-9261-1e9577f8c46a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.888600 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.888576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27333cea-cd69-46ed-9261-1e9577f8c46a-config-out\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.888704 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.888643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-web-config\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.888764 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.888730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.888918 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.888883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.888994 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.888969 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.889505 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.889483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.891044 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.891024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27333cea-cd69-46ed-9261-1e9577f8c46a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:39.891347 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:39.891324 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2c6m\" (UniqueName: \"kubernetes.io/projected/27333cea-cd69-46ed-9261-1e9577f8c46a-kube-api-access-l2c6m\") pod \"prometheus-k8s-0\" (UID: \"27333cea-cd69-46ed-9261-1e9577f8c46a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:40.028727 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:40.028638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:40.582208 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:40.582175 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 15:56:40.582757 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:56:40.582732 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27333cea_cd69_46ed_9261_1e9577f8c46a.slice/crio-bb25157300966f6366487ca11771204765957e66ef084d8086f0062f7de81276 WatchSource:0}: Error finding container bb25157300966f6366487ca11771204765957e66ef084d8086f0062f7de81276: Status 404 returned error can't find the container with id bb25157300966f6366487ca11771204765957e66ef084d8086f0062f7de81276 Sep 09 15:56:41.171797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.171762 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"7cb43134388e0d49444eeb989253380459962e147e9b0e299e800ec661c49c1c"} Sep 09 15:56:41.172173 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.171873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"1682e7a57190153999ba66496719f74e976f776eccbe05ec0ab1c32b47725906"} Sep 09 15:56:41.172173 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.171906 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"298aa1b12bb7866e35bf059d6ca9e53aa4f64d0e1b6bc1861306b4be0c345e9c"} Sep 09 15:56:41.173271 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.173241 2562 generic.go:356] "Generic (PLEG): container finished" podID="27333cea-cd69-46ed-9261-1e9577f8c46a" containerID="d19a68ae8f4ec193affafcd9cdcd4c26c10b9c69eec9733602ee5a83e2dae02b" exitCode=0 Sep 09 15:56:41.173420 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.173284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerDied","Data":"d19a68ae8f4ec193affafcd9cdcd4c26c10b9c69eec9733602ee5a83e2dae02b"} Sep 09 15:56:41.173420 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.173345 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"bb25157300966f6366487ca11771204765957e66ef084d8086f0062f7de81276"} Sep 09 15:56:41.175089 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.175070 2562 generic.go:356] "Generic (PLEG): container finished" podID="66001ff9-94de-4765-88f4-450c64c01efc" containerID="6a357e7cef7705cacdea29649428829502f5ac8dfafee11b0a62957bb1b38b07" exitCode=0 Sep 09 15:56:41.175182 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.175155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qk4f" event={"ID":"66001ff9-94de-4765-88f4-450c64c01efc","Type":"ContainerDied","Data":"6a357e7cef7705cacdea29649428829502f5ac8dfafee11b0a62957bb1b38b07"} Sep 09 15:56:41.178596 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.178579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"800d498051ceb3157c14558651dc6e8d77ef6c4046aef6c59cda4e373d3ed339"} Sep 09 15:56:41.178673 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.178631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"53f86c11e58861d8a3a5d4ef77c8d30a98e83f4095e3e3774ba06cf2dd325677"} Sep 09 15:56:41.178673 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.178658 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"ff19dfc4379970b12c4220783382d8548a8f34f4d5b91ecece310929d7cba81f"} Sep 09 15:56:41.178744 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.178680 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"540e2f64c6d4d24f2c595441ca26c5d56c3e5a1ae18fc9639dd314c4c6ae9e3e"} Sep 09 15:56:41.178744 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:41.178705 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"f4279600537c3b390a75fc26530fdd14dc7289e6af1854f5ead1287ac21b484b"} Sep 09 15:56:42.187768 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:42.187317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qk4f" event={"ID":"66001ff9-94de-4765-88f4-450c64c01efc","Type":"ContainerStarted","Data":"2a753ebe1de1a07226f5035d46671d60c93acb962fd08f79bd1d150593f17908"} Sep 09 15:56:42.187768 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:42.187374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qk4f" event={"ID":"66001ff9-94de-4765-88f4-450c64c01efc","Type":"ContainerStarted","Data":"cab493f89fde1f4f9799d886014a33cc71b876303db792d16fb5a189d5687f54"} Sep 09 15:56:42.205335 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:42.205279 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2qk4f" podStartSLOduration=3.297605252 podStartE2EDuration="9.205256292s" totalImagesPullingTime="5.824368656s" totalInitContainerRuntime="83.282384ms" isStatefulPod=true podCreationTimestamp="2026-09-09 15:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:42.204392609 +0000 UTC m=+321.545584441" watchObservedRunningTime="2026-09-09 15:56:42.205256292 +0000 UTC m=+321.546448133" Sep 09 15:56:43.195363 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.195320 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"1a9100bd51a5c20dfa97b43323d4beb7cc7313666c2790239e53610529f5b837"} Sep 09 15:56:43.195837 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.195442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"a17fd4afa7de131a1c01e50409f2592c610cc3b9f99f0e1979ea5f41c833dc41"} Sep 09 15:56:43.195837 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.195486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" event={"ID":"215b1af2-8686-45db-a869-efb1ab5009b3","Type":"ContainerStarted","Data":"70cf8435bdf8e8ab8b7a1c6308e2d0a165709636760351177814457dfee07383"} Sep 09 15:56:43.195837 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.195525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:43.199659 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.199628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9069f88-8c0c-46bf-8d0c-cfa299f54513","Type":"ContainerStarted","Data":"97b1cb9e26f2f9abc0f4f81da02288c7fe54fea7acfdcff0ca1b61e4340bbdd4"} Sep 09 15:56:43.222756 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.222698 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" podStartSLOduration=2.3595490359999998 podStartE2EDuration="7.222675684s" totalImagesPullingTime="4.863126648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:43.221344599 +0000 UTC m=+322.562536439" watchObservedRunningTime="2026-09-09 15:56:43.222675684 +0000 UTC m=+322.563867514" Sep 09 15:56:43.250962 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:43.250900 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.225010798 podStartE2EDuration="9.250876725s" totalImagesPullingTime="6.00337855s" totalInitContainerRuntime="22.487377ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:43.249560588 +0000 UTC m=+322.590752419" watchObservedRunningTime="2026-09-09 15:56:43.250876725 +0000 UTC m=+322.592068565" Sep 09 15:56:45.211929 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:45.211901 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"38ff14752af8da3e815c4f1771c8e0c9d0411a49c4159d6dad00cff6e462933a"} Sep 09 15:56:46.219385 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.219350 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"eecabc41e316f998937a6980bf182e2bc69304c3e1faf50032338cd980c4a6ff"} Sep 09 15:56:46.219859 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.219457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"44126e8fd96c74d8a948782bef496b6881bff35a616c182f2456a3d66224360d"} Sep 09 15:56:46.219859 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.219486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"0689c2c9f0d0387107f711af51c1f03d45a6cb25c6cdfaf6586e2ac0ac719812"} Sep 09 15:56:46.219859 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.219517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"917613cf7d48167096f7ca0ac30cc1cd48064a71aa59a7b478fcabfa0e93c8db"} Sep 09 15:56:46.219859 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.219541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27333cea-cd69-46ed-9261-1e9577f8c46a","Type":"ContainerStarted","Data":"b70b01187553c479ef4ddd8ce090d7664da6b5857d4986029d201c9a8d1a95cd"} Sep 09 15:56:46.250191 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:46.250099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.385740335 podStartE2EDuration="7.250085314s" totalImagesPullingTime="3.847458248s" totalInitContainerRuntime="16.886731ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:46.248800749 +0000 UTC m=+325.589992578" watchObservedRunningTime="2026-09-09 15:56:46.250085314 +0000 UTC m=+325.591277143" Sep 09 15:56:49.211391 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:49.211360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75b97b9786-k62bs" Sep 09 15:56:50.029159 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:50.029121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:56:52.240321 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.240276 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerName="registry" containerID="cri-o://964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5" gracePeriod=30 Sep 09 15:56:52.479556 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.479534 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:52.603805 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603775 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.603805 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603810 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603867 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603927 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmcft\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604035 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.603968 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token\") pod \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\" (UID: \"79f9c5e2-bb0b-498d-82db-a203c5a1feea\") " Sep 09 15:56:52.604336 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.604301 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:56:52.604616 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.604569 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:56:52.606241 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.606213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:56:52.606495 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.606447 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:56:52.606609 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.606577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:56:52.606609 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.606596 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft" (OuterVolumeSpecName: "kube-api-access-bmcft") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "kube-api-access-bmcft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:56:52.607033 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.607016 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:56:52.612590 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.612567 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "79f9c5e2-bb0b-498d-82db-a203c5a1feea" (UID: "79f9c5e2-bb0b-498d-82db-a203c5a1feea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:56:52.705249 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705220 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-installation-pull-secrets\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705249 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705247 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmcft\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-kube-api-access-bmcft\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705260 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-bound-sa-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705274 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f9c5e2-bb0b-498d-82db-a203c5a1feea-image-registry-private-configuration\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705287 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f9c5e2-bb0b-498d-82db-a203c5a1feea-ca-trust-extracted\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705299 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-certificates\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705311 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f9c5e2-bb0b-498d-82db-a203c5a1feea-registry-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:52.705436 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:52.705322 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f9c5e2-bb0b-498d-82db-a203c5a1feea-trusted-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:53.248736 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.248702 2562 generic.go:356] "Generic (PLEG): container finished" podID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerID="964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5" exitCode=0 Sep 09 15:56:53.249245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.248761 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" Sep 09 15:56:53.249245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.248791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" event={"ID":"79f9c5e2-bb0b-498d-82db-a203c5a1feea","Type":"ContainerDied","Data":"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5"} Sep 09 15:56:53.249245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.248859 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5494898f59-lqwlc" event={"ID":"79f9c5e2-bb0b-498d-82db-a203c5a1feea","Type":"ContainerDied","Data":"8a163d939e17536ba60bc87bbc22a09197a21f5310c4ba0c22ca4213b0ea2d10"} Sep 09 15:56:53.249245 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.248894 2562 scope.go:117] "RemoveContainer" containerID="964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5" Sep 09 15:56:53.250551 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.250515 2562 generic.go:356] "Generic (PLEG): container finished" podID="e4f87f05-7a3b-4ca1-b9cd-49d10dabf520" containerID="c405ac86ea2b9e8cb79341281484d04b4d5439f67cc57640f2e385b563beae32" exitCode=0 Sep 09 15:56:53.250637 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.250554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" event={"ID":"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520","Type":"ContainerDied","Data":"c405ac86ea2b9e8cb79341281484d04b4d5439f67cc57640f2e385b563beae32"} Sep 09 15:56:53.250841 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.250828 2562 scope.go:117] "RemoveContainer" containerID="c405ac86ea2b9e8cb79341281484d04b4d5439f67cc57640f2e385b563beae32" Sep 09 15:56:53.260431 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.260385 2562 scope.go:117] "RemoveContainer" containerID="964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5" Sep 09 15:56:53.260687 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:53.260668 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5\": container with ID starting with 964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5 not found: ID does not exist" containerID="964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5" Sep 09 15:56:53.260760 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.260695 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5"} err="failed to get container status \"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5\": rpc error: code = NotFound desc = could not find container \"964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5\": container with ID starting with 964133721e815f67947d42cdc44e52dddfe9847ef4b4c780506fe6acaacd61d5 not found: ID does not exist" Sep 09 15:56:53.284257 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.284228 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:56:53.290489 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:53.290461 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5494898f59-lqwlc"] Sep 09 15:56:54.263816 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:54.263778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-gwb7t" event={"ID":"e4f87f05-7a3b-4ca1-b9cd-49d10dabf520","Type":"ContainerStarted","Data":"6ed51d21d8711314edb7bb84681594bf8c119959c40a9630cc9ab2d67df5fdbd"} Sep 09 15:56:55.189092 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:55.189057 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" path="/var/lib/kubelet/pods/79f9c5e2-bb0b-498d-82db-a203c5a1feea/volumes" Sep 09 15:56:57.346529 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.346469 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c9fb5668-ssjkr" podUID="31ac3678-1c2e-4578-b39d-0645b23677ef" containerName="console" containerID="cri-o://2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0" gracePeriod=15 Sep 09 15:56:57.637215 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.637181 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:57.748343 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748311 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748528 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748376 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpntk\" (UniqueName: \"kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748528 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748402 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748528 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748446 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748528 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748500 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748528 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748518 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config\") pod \"31ac3678-1c2e-4578-b39d-0645b23677ef\" (UID: \"31ac3678-1c2e-4578-b39d-0645b23677ef\") " Sep 09 15:56:57.748910 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:56:57.748910 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748890 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:56:57.749099 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.748963 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config" (OuterVolumeSpecName: "console-config") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 15:56:57.750826 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.750794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:56:57.750916 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.750826 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 15:56:57.750916 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.750847 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk" (OuterVolumeSpecName: "kube-api-access-xpntk") pod "31ac3678-1c2e-4578-b39d-0645b23677ef" (UID: "31ac3678-1c2e-4578-b39d-0645b23677ef"). InnerVolumeSpecName "kube-api-access-xpntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:56:57.849665 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849630 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:57.849665 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849659 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpntk\" (UniqueName: \"kubernetes.io/projected/31ac3678-1c2e-4578-b39d-0645b23677ef-kube-api-access-xpntk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:57.849665 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849668 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-oauth-config\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:57.849928 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849678 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-oauth-serving-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:57.849928 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849691 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31ac3678-1c2e-4578-b39d-0645b23677ef-console-serving-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:57.849928 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:57.849700 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31ac3678-1c2e-4578-b39d-0645b23677ef-console-config\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 15:56:58.282304 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.282274 2562 generic.go:356] "Generic (PLEG): container finished" podID="31ac3678-1c2e-4578-b39d-0645b23677ef" containerID="2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0" exitCode=0 Sep 09 15:56:58.282496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.282320 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c9fb5668-ssjkr" Sep 09 15:56:58.282496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.282361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c9fb5668-ssjkr" event={"ID":"31ac3678-1c2e-4578-b39d-0645b23677ef","Type":"ContainerDied","Data":"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0"} Sep 09 15:56:58.282496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.282447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c9fb5668-ssjkr" event={"ID":"31ac3678-1c2e-4578-b39d-0645b23677ef","Type":"ContainerDied","Data":"870f8eec50ec645ed154fb5c0c4e014cb5462350e7bbca0e39856ca085b65c71"} Sep 09 15:56:58.282496 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.282473 2562 scope.go:117] "RemoveContainer" containerID="2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0" Sep 09 15:56:58.290888 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.290868 2562 scope.go:117] "RemoveContainer" containerID="2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0" Sep 09 15:56:58.291137 ip-10-0-130-57 kubenswrapper[2562]: E0909 15:56:58.291117 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0\": container with ID starting with 2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0 not found: ID does not exist" containerID="2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0" Sep 09 15:56:58.291188 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.291146 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0"} err="failed to get container status \"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0\": rpc error: code = NotFound desc = could not find container \"2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0\": container with ID starting with 2185e78647c07106869f3bf0a2589ecd6840930b930c34a91297e78dbccac8f0 not found: ID does not exist" Sep 09 15:56:58.303308 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.303288 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:58.306958 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:58.306938 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c9fb5668-ssjkr"] Sep 09 15:56:59.189194 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:56:59.189162 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ac3678-1c2e-4578-b39d-0645b23677ef" path="/var/lib/kubelet/pods/31ac3678-1c2e-4578-b39d-0645b23677ef/volumes" Sep 09 15:57:35.162276 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:35.162236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:57:35.164507 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:35.164488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337bfd9d-63ad-43b9-9b6f-e79aab96a48f-metrics-certs\") pod \"network-metrics-daemon-wpwhx\" (UID: \"337bfd9d-63ad-43b9-9b6f-e79aab96a48f\") " pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:57:35.287322 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:35.287296 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2dhs\"" Sep 09 15:57:35.295531 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:35.295515 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpwhx" Sep 09 15:57:35.562128 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:35.562051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpwhx"] Sep 09 15:57:35.564881 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:57:35.564855 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337bfd9d_63ad_43b9_9b6f_e79aab96a48f.slice/crio-a9403c8fc4e44b1f079902d78f4661241bc3cc497cc0ed8feda5f7ca26b948cf WatchSource:0}: Error finding container a9403c8fc4e44b1f079902d78f4661241bc3cc497cc0ed8feda5f7ca26b948cf: Status 404 returned error can't find the container with id a9403c8fc4e44b1f079902d78f4661241bc3cc497cc0ed8feda5f7ca26b948cf Sep 09 15:57:36.442809 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:36.442774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpwhx" event={"ID":"337bfd9d-63ad-43b9-9b6f-e79aab96a48f","Type":"ContainerStarted","Data":"a9403c8fc4e44b1f079902d78f4661241bc3cc497cc0ed8feda5f7ca26b948cf"} Sep 09 15:57:38.452508 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:38.452473 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpwhx" event={"ID":"337bfd9d-63ad-43b9-9b6f-e79aab96a48f","Type":"ContainerStarted","Data":"afc89789d17298d8a1c808d91af6b6cd5a8670ffba6456658537d31f35dccdeb"} Sep 09 15:57:38.452896 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:38.452526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpwhx" event={"ID":"337bfd9d-63ad-43b9-9b6f-e79aab96a48f","Type":"ContainerStarted","Data":"bd0a450432cc2986cb460b6e6f1dc7afb32e3a319977b321c80c7d5a93fb01eb"} Sep 09 15:57:38.469991 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:38.469942 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wpwhx" podStartSLOduration=375.167317561 podStartE2EDuration="6m17.469924866s" totalImagesPullingTime="2.302607305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:38.469099712 +0000 UTC m=+377.810291542" watchObservedRunningTime="2026-09-09 15:57:38.469924866 +0000 UTC m=+377.811116694" Sep 09 15:57:40.029249 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:40.029209 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:57:40.048068 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:40.048042 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:57:40.475264 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:40.475236 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 15:57:57.711931 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.711887 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz"] Sep 09 15:57:57.712562 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712383 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerName="registry" Sep 09 15:57:57.712562 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerName="registry" Sep 09 15:57:57.712562 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712443 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ac3678-1c2e-4578-b39d-0645b23677ef" containerName="console" Sep 09 15:57:57.712562 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712452 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ac3678-1c2e-4578-b39d-0645b23677ef" containerName="console" Sep 09 15:57:57.712797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712565 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f9c5e2-bb0b-498d-82db-a203c5a1feea" containerName="registry" Sep 09 15:57:57.712797 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.712579 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ac3678-1c2e-4578-b39d-0645b23677ef" containerName="console" Sep 09 15:57:57.715148 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.715126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.717807 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717771 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 15:57:57.717942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717806 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 15:57:57.717942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717771 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 15:57:57.717942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717778 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 15:57:57.717942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717852 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qmlw4\"" Sep 09 15:57:57.717942 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.717809 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 15:57:57.722524 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.722506 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 15:57:57.727288 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.727266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz"] Sep 09 15:57:57.856308 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856318 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-serving-certs-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5r2\" (UniqueName: \"kubernetes.io/projected/d194cb63-9e92-49de-b17a-c3ce77797d16-kube-api-access-4f5r2\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-metrics-client-ca\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856533 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-federate-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856772 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.856772 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.856603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957212 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-serving-certs-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5r2\" (UniqueName: \"kubernetes.io/projected/d194cb63-9e92-49de-b17a-c3ce77797d16-kube-api-access-4f5r2\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-metrics-client-ca\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957440 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-federate-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957729 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.957729 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.957489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.958080 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.958052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-serving-certs-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.958181 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.958093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-metrics-client-ca\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.958304 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.958274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.959895 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.959876 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-federate-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.960068 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.960046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.960111 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.960102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-secret-telemeter-client\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.960192 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.960172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d194cb63-9e92-49de-b17a-c3ce77797d16-telemeter-client-tls\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:57.965841 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:57.965783 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5r2\" (UniqueName: \"kubernetes.io/projected/d194cb63-9e92-49de-b17a-c3ce77797d16-kube-api-access-4f5r2\") pod \"telemeter-client-b99c5c8f9-bp9wz\" (UID: \"d194cb63-9e92-49de-b17a-c3ce77797d16\") " pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:58.025344 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:58.025308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" Sep 09 15:57:58.302356 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:58.302326 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz"] Sep 09 15:57:58.303457 ip-10-0-130-57 kubenswrapper[2562]: W0909 15:57:58.303426 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd194cb63_9e92_49de_b17a_c3ce77797d16.slice/crio-7f0b191a95a17e7f10729b6cee06a84ad502d5a44761c40a80bf6796722aaa25 WatchSource:0}: Error finding container 7f0b191a95a17e7f10729b6cee06a84ad502d5a44761c40a80bf6796722aaa25: Status 404 returned error can't find the container with id 7f0b191a95a17e7f10729b6cee06a84ad502d5a44761c40a80bf6796722aaa25 Sep 09 15:57:58.531867 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:57:58.531832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" event={"ID":"d194cb63-9e92-49de-b17a-c3ce77797d16","Type":"ContainerStarted","Data":"7f0b191a95a17e7f10729b6cee06a84ad502d5a44761c40a80bf6796722aaa25"} Sep 09 15:58:00.542205 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:58:00.542174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" event={"ID":"d194cb63-9e92-49de-b17a-c3ce77797d16","Type":"ContainerStarted","Data":"59443efe39ada7b613d675798f06fb94a06ac9741c1201968ca80794be078224"} Sep 09 15:58:00.542547 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:58:00.542280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" event={"ID":"d194cb63-9e92-49de-b17a-c3ce77797d16","Type":"ContainerStarted","Data":"4cf7070fb2fcc3ab5c4b7a7de1e5516eaf5a79d708ded4a2ae99ca40e1d07a74"} Sep 09 15:58:01.548869 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:58:01.548836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" event={"ID":"d194cb63-9e92-49de-b17a-c3ce77797d16","Type":"ContainerStarted","Data":"1f946468e0933e6dccd7a7df0bdcf336170b3363df0ad83034d15937cf133c33"} Sep 09 15:58:01.570839 ip-10-0-130-57 kubenswrapper[2562]: I0909 15:58:01.570789 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b99c5c8f9-bp9wz" podStartSLOduration=2.49200509 podStartE2EDuration="4.570769237s" totalImagesPullingTime="2.078764147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:58:01.568404046 +0000 UTC m=+400.909595875" watchObservedRunningTime="2026-09-09 15:58:01.570769237 +0000 UTC m=+400.911961068" Sep 09 16:00:49.167748 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.167712 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lcxpg"] Sep 09 16:00:49.170996 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.170976 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.173660 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.173637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 16:00:49.173832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.173813 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6km9v\"" Sep 09 16:00:49.174891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.174871 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 16:00:49.180476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.180454 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lcxpg"] Sep 09 16:00:49.279645 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.279606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.279808 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.279653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2cj\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-kube-api-access-tf2cj\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.380879 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.380843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.381077 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.380894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2cj\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-kube-api-access-tf2cj\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.390261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.390234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.390445 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.390427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2cj\" (UniqueName: \"kubernetes.io/projected/27b46b27-90fb-4696-9542-577514e1afed-kube-api-access-tf2cj\") pod \"cert-manager-webhook-6486ffd997-lcxpg\" (UID: \"27b46b27-90fb-4696-9542-577514e1afed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.492771 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.492682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:49.760354 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:49.760243 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lcxpg"] Sep 09 16:00:49.763939 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:00:49.763916 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27b46b27_90fb_4696_9542_577514e1afed.slice/crio-93e4043e0a4ba49161a9f54ed512a2e76b98ab443384058385f01a1d0e5d729f WatchSource:0}: Error finding container 93e4043e0a4ba49161a9f54ed512a2e76b98ab443384058385f01a1d0e5d729f: Status 404 returned error can't find the container with id 93e4043e0a4ba49161a9f54ed512a2e76b98ab443384058385f01a1d0e5d729f Sep 09 16:00:50.226596 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:50.226564 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" event={"ID":"27b46b27-90fb-4696-9542-577514e1afed","Type":"ContainerStarted","Data":"93e4043e0a4ba49161a9f54ed512a2e76b98ab443384058385f01a1d0e5d729f"} Sep 09 16:00:53.242871 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:53.242779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" event={"ID":"27b46b27-90fb-4696-9542-577514e1afed","Type":"ContainerStarted","Data":"391142b5d20cd99571bcf6e00bd8562bd89727f0f0c14af92dc1b5e4427c40f6"} Sep 09 16:00:53.243237 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:53.242914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:00:53.263258 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:53.263204 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" podStartSLOduration=1.143530024 podStartE2EDuration="4.263187695s" totalImagesPullingTime="3.119657671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:00:53.262275002 +0000 UTC m=+572.603466832" watchObservedRunningTime="2026-09-09 16:00:53.263187695 +0000 UTC m=+572.604379525" Sep 09 16:00:59.249017 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:00:59.248985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lcxpg" Sep 09 16:01:19.181727 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.181684 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7"] Sep 09 16:01:19.185980 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.185945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.189789 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.189766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 16:01:19.190312 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.190296 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jndt2\"" Sep 09 16:01:19.190598 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.190582 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 16:01:19.193127 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.192892 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 16:01:19.194056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.194037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 16:01:19.212746 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.212720 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7"] Sep 09 16:01:19.220971 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.220947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.221072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.220981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pkf\" (UniqueName: \"kubernetes.io/projected/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-kube-api-access-j6pkf\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.221072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.221030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.221072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.221064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.221177 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.221080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pkf\" (UniqueName: \"kubernetes.io/projected/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-kube-api-access-j6pkf\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.322843 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.322808 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-manifests\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.324793 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.324758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-tmp\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.324920 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.324897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.326048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.326019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.331378 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.331356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pkf\" (UniqueName: \"kubernetes.io/projected/f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3-kube-api-access-j6pkf\") pod \"opendatahub-operator-controller-manager-b4ccd886c-d5vn7\" (UID: \"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3\") " pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.496329 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.496244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:19.773862 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:19.773782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7"] Sep 09 16:01:19.777611 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:01:19.777583 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ccdb23_9f94_4c75_9d91_5be4e1ac66b3.slice/crio-8004f40b3d1b01d56e2fe90b72beae9f4741d6ad4cbc03a3e6ea2cf6aa98f0a4 WatchSource:0}: Error finding container 8004f40b3d1b01d56e2fe90b72beae9f4741d6ad4cbc03a3e6ea2cf6aa98f0a4: Status 404 returned error can't find the container with id 8004f40b3d1b01d56e2fe90b72beae9f4741d6ad4cbc03a3e6ea2cf6aa98f0a4 Sep 09 16:01:20.352951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:20.352907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" event={"ID":"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3","Type":"ContainerStarted","Data":"8004f40b3d1b01d56e2fe90b72beae9f4741d6ad4cbc03a3e6ea2cf6aa98f0a4"} Sep 09 16:01:23.369710 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:23.369673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" event={"ID":"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3","Type":"ContainerStarted","Data":"85d04d6aa40f16c756b3e8f93a06566d549654f1d65dd1193d992aebd6c57929"} Sep 09 16:01:23.609705 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:23.609680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:01:23.611891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:23.609712 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:01:24.375391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:24.375354 2562 generic.go:356] "Generic (PLEG): container finished" podID="f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3" containerID="85d04d6aa40f16c756b3e8f93a06566d549654f1d65dd1193d992aebd6c57929" exitCode=0 Sep 09 16:01:24.375835 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:24.375447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" event={"ID":"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3","Type":"ContainerDied","Data":"85d04d6aa40f16c756b3e8f93a06566d549654f1d65dd1193d992aebd6c57929"} Sep 09 16:01:24.375903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:24.375887 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 16:01:25.382717 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:25.382686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" event={"ID":"f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3","Type":"ContainerStarted","Data":"31bdda7c35df70dd746aafc7892ce6cc96b6d35d51802cb392168658671ceaa9"} Sep 09 16:01:25.383101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:25.382809 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:25.404265 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:25.404203 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" podStartSLOduration=1.8319650109999999 podStartE2EDuration="6.404181728s" totalImagesPullingTime="3.520047982s" totalInitContainerRuntime="1.052168735s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:25.402219094 +0000 UTC m=+604.743410927" watchObservedRunningTime="2026-09-09 16:01:25.404181728 +0000 UTC m=+604.745373558" Sep 09 16:01:36.390100 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:36.390068 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b4ccd886c-d5vn7" Sep 09 16:01:50.519459 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.519424 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf"] Sep 09 16:01:50.523039 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.523022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.528004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.527978 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 16:01:50.528126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.528006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9mscs\"" Sep 09 16:01:50.528126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.527978 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 16:01:50.528126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.528043 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 16:01:50.542686 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.542664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf"] Sep 09 16:01:50.700051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c70978-568b-4536-934d-632711d3636c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbdj\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-kube-api-access-kcbdj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700443 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.700443 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.700292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c70978-568b-4536-934d-632711d3636c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbdj\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-kube-api-access-kcbdj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c70978-568b-4536-934d-632711d3636c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c70978-568b-4536-934d-632711d3636c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801981 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.801981 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801880 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.802085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.801998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.802329 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.802305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.802579 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.802556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.802675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.802660 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c70978-568b-4536-934d-632711d3636c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.803916 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.803892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c70978-568b-4536-934d-632711d3636c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.804338 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.804319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c70978-568b-4536-934d-632711d3636c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.813052 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.813020 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.813271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.813249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbdj\" (UniqueName: \"kubernetes.io/projected/c2c70978-568b-4536-934d-632711d3636c-kube-api-access-kcbdj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf\" (UID: \"c2c70978-568b-4536-934d-632711d3636c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:50.833176 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:50.833143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:51.118818 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.118789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf"] Sep 09 16:01:51.120430 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:01:51.120377 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c70978_568b_4536_934d_632711d3636c.slice/crio-a56cf453b0dcb20cd6a1326f6e39bb6e8788afe7c495a641c6c54516d31d91a7 WatchSource:0}: Error finding container a56cf453b0dcb20cd6a1326f6e39bb6e8788afe7c495a641c6c54516d31d91a7: Status 404 returned error can't find the container with id a56cf453b0dcb20cd6a1326f6e39bb6e8788afe7c495a641c6c54516d31d91a7 Sep 09 16:01:51.504943 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.504857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" event={"ID":"c2c70978-568b-4536-934d-632711d3636c","Type":"ContainerStarted","Data":"a56cf453b0dcb20cd6a1326f6e39bb6e8788afe7c495a641c6c54516d31d91a7"} Sep 09 16:01:51.893974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.893935 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w4nn8"] Sep 09 16:01:51.906305 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.906267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:51.909221 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.909194 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bjxsq\"" Sep 09 16:01:51.909221 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.909212 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 09 16:01:51.912489 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.912465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:51.912684 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.912661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77m5\" (UniqueName: \"kubernetes.io/projected/8c401f4f-6576-4d25-aa7f-4e299da3de77-kube-api-access-k77m5\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:51.916961 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:51.916933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w4nn8"] Sep 09 16:01:52.013337 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.013294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77m5\" (UniqueName: \"kubernetes.io/projected/8c401f4f-6576-4d25-aa7f-4e299da3de77-kube-api-access-k77m5\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:52.013524 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.013397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:52.013524 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:01:52.013512 2562 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 09 16:01:52.013599 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:01:52.013574 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs podName:8c401f4f-6576-4d25-aa7f-4e299da3de77 nodeName:}" failed. No retries permitted until 2026-09-09 16:01:52.513554753 +0000 UTC m=+631.854746575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs") pod "model-serving-api-54d89bd5fc-w4nn8" (UID: "8c401f4f-6576-4d25-aa7f-4e299da3de77") : secret "model-serving-api-tls" not found Sep 09 16:01:52.023877 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.023837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77m5\" (UniqueName: \"kubernetes.io/projected/8c401f4f-6576-4d25-aa7f-4e299da3de77-kube-api-access-k77m5\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:52.519290 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.519249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:52.523032 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.522972 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c401f4f-6576-4d25-aa7f-4e299da3de77-tls-certs\") pod \"model-serving-api-54d89bd5fc-w4nn8\" (UID: \"8c401f4f-6576-4d25-aa7f-4e299da3de77\") " pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:52.818265 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:52.818170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:53.104083 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:53.104050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w4nn8"] Sep 09 16:01:53.105527 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:01:53.105494 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c401f4f_6576_4d25_aa7f_4e299da3de77.slice/crio-3c3891e3283ded4fd6e45dc62ea91176598dec537e4e97f8d11ce2f270d76c4e WatchSource:0}: Error finding container 3c3891e3283ded4fd6e45dc62ea91176598dec537e4e97f8d11ce2f270d76c4e: Status 404 returned error can't find the container with id 3c3891e3283ded4fd6e45dc62ea91176598dec537e4e97f8d11ce2f270d76c4e Sep 09 16:01:53.519814 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:53.519728 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" event={"ID":"8c401f4f-6576-4d25-aa7f-4e299da3de77","Type":"ContainerStarted","Data":"3c3891e3283ded4fd6e45dc62ea91176598dec537e4e97f8d11ce2f270d76c4e"} Sep 09 16:01:54.230988 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.230949 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:01:54.231260 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.231023 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:01:54.231260 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.231052 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:01:54.531059 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.530918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" event={"ID":"c2c70978-568b-4536-934d-632711d3636c","Type":"ContainerStarted","Data":"aa598f192f9cecc629fe88c43bf003229dd222bf3ee90b603e74a65e4f3eea3c"} Sep 09 16:01:54.554602 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.553306 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" podStartSLOduration=1.445222358 podStartE2EDuration="4.553285666s" totalImagesPullingTime="3.108063308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:54.551152397 +0000 UTC m=+633.892344230" watchObservedRunningTime="2026-09-09 16:01:54.553285666 +0000 UTC m=+633.894477494" Sep 09 16:01:54.833587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:54.833494 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:55.838444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:55.838390 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:56.541550 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:56.541515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:56.542525 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:56.542502 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf" Sep 09 16:01:57.548426 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:57.548366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" event={"ID":"8c401f4f-6576-4d25-aa7f-4e299da3de77","Type":"ContainerStarted","Data":"f97fa436043ad28f27097cba4ec507dcd1407d7f983c13e919036fc725fc1033"} Sep 09 16:01:57.548817 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:57.548448 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:01:57.566208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:01:57.566153 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" podStartSLOduration=3.038873326 podStartE2EDuration="6.56613823s" totalImagesPullingTime="3.527264904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:01:57.565363855 +0000 UTC m=+636.906555685" watchObservedRunningTime="2026-09-09 16:01:57.56613823 +0000 UTC m=+636.907330058" Sep 09 16:02:00.738367 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.738331 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r"] Sep 09 16:02:00.740653 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.740628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.743273 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.743240 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 16:02:00.745959 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.745934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 16:02:00.746115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.746097 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 16:02:00.746605 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.746588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hs9zx\"" Sep 09 16:02:00.746684 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.746593 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 16:02:00.746684 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.746624 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 16:02:00.755582 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.755558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r"] Sep 09 16:02:00.804033 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.804008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-manager-config\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.804156 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.804049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmx47\" (UniqueName: \"kubernetes.io/projected/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-kube-api-access-rmx47\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.804156 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.804081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.804231 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.804207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.905150 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.905115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.905150 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.905157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-manager-config\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.905424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.905180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmx47\" (UniqueName: \"kubernetes.io/projected/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-kube-api-access-rmx47\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.905424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.905205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.905932 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.905908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-manager-config\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.907715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.907696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.907800 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.907728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-metrics-cert\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:00.918531 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:00.918504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmx47\" (UniqueName: \"kubernetes.io/projected/2b7067f9-9854-4b29-81cf-d6f41c4da5d4-kube-api-access-rmx47\") pod \"lws-controller-manager-678f94cf5f-h886r\" (UID: \"2b7067f9-9854-4b29-81cf-d6f41c4da5d4\") " pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:01.049635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:01.049540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:01.333593 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:01.333562 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r"] Sep 09 16:02:01.334429 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:02:01.334379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7067f9_9854_4b29_81cf_d6f41c4da5d4.slice/crio-d3d29037b3e9a4df25f7e87fbe83763639d3bd32efa512ac6486e0e6b6bb0d33 WatchSource:0}: Error finding container d3d29037b3e9a4df25f7e87fbe83763639d3bd32efa512ac6486e0e6b6bb0d33: Status 404 returned error can't find the container with id d3d29037b3e9a4df25f7e87fbe83763639d3bd32efa512ac6486e0e6b6bb0d33 Sep 09 16:02:01.570319 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:01.570286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" event={"ID":"2b7067f9-9854-4b29-81cf-d6f41c4da5d4","Type":"ContainerStarted","Data":"d3d29037b3e9a4df25f7e87fbe83763639d3bd32efa512ac6486e0e6b6bb0d33"} Sep 09 16:02:02.355004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.354962 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-scjp2"] Sep 09 16:02:02.357823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.357801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.365327 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.365307 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 16:02:02.369092 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.369076 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nmbqq\"" Sep 09 16:02:02.380779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.380754 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-scjp2"] Sep 09 16:02:02.382153 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.382131 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh"] Sep 09 16:02:02.384476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.384456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:02.387617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.387578 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m57rx\"" Sep 09 16:02:02.387813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.387795 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 16:02:02.405366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.405342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh"] Sep 09 16:02:02.420373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.420346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzj4s\" (UniqueName: \"kubernetes.io/projected/3cdc679a-6c26-406b-8223-49f2a4c44b06-kube-api-access-jzj4s\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.420495 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.420403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.521027 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.521002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzj4s\" (UniqueName: \"kubernetes.io/projected/3cdc679a-6c26-406b-8223-49f2a4c44b06-kube-api-access-jzj4s\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.521162 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.521040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.521162 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.521090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:02.521162 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.521106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmlc\" (UniqueName: \"kubernetes.io/projected/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-kube-api-access-sgmlc\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:02.521259 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:02:02.521205 2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 16:02:02.521292 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:02:02.521270 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert podName:3cdc679a-6c26-406b-8223-49f2a4c44b06 nodeName:}" failed. No retries permitted until 2026-09-09 16:02:03.021250519 +0000 UTC m=+642.362442342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert") pod "kserve-controller-manager-d6f4b65b8-scjp2" (UID: "3cdc679a-6c26-406b-8223-49f2a4c44b06") : secret "kserve-webhook-server-cert" not found Sep 09 16:02:02.546866 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.546843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzj4s\" (UniqueName: \"kubernetes.io/projected/3cdc679a-6c26-406b-8223-49f2a4c44b06-kube-api-access-jzj4s\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:02.622430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.622351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:02.622430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.622382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmlc\" (UniqueName: \"kubernetes.io/projected/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-kube-api-access-sgmlc\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:02.622562 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:02:02.622516 2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 09 16:02:02.622597 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:02:02.622572 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert podName:fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5 nodeName:}" failed. No retries permitted until 2026-09-09 16:02:03.122555763 +0000 UTC m=+642.463747584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert") pod "llmisvc-controller-manager-777c6bd7d5-zmlrh" (UID: "fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5") : secret "llmisvc-webhook-server-cert" not found Sep 09 16:02:02.662572 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:02.662538 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmlc\" (UniqueName: \"kubernetes.io/projected/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-kube-api-access-sgmlc\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:03.025682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.025600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:03.027993 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.027970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdc679a-6c26-406b-8223-49f2a4c44b06-cert\") pod \"kserve-controller-manager-d6f4b65b8-scjp2\" (UID: \"3cdc679a-6c26-406b-8223-49f2a4c44b06\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:03.126039 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.126005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:03.128311 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.128291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zmlrh\" (UID: \"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:03.267863 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.267825 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:03.295617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.295588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:03.565283 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:02:03.565258 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdc679a_6c26_406b_8223_49f2a4c44b06.slice/crio-8caf78981a551b0af8d6653358b83a8580b1be109c6042a02d1b4db4c3056cd1 WatchSource:0}: Error finding container 8caf78981a551b0af8d6653358b83a8580b1be109c6042a02d1b4db4c3056cd1: Status 404 returned error can't find the container with id 8caf78981a551b0af8d6653358b83a8580b1be109c6042a02d1b4db4c3056cd1 Sep 09 16:02:03.568288 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.568265 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-scjp2"] Sep 09 16:02:03.580702 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.580675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" event={"ID":"3cdc679a-6c26-406b-8223-49f2a4c44b06","Type":"ContainerStarted","Data":"8caf78981a551b0af8d6653358b83a8580b1be109c6042a02d1b4db4c3056cd1"} Sep 09 16:02:03.598060 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:03.598037 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh"] Sep 09 16:02:03.599855 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:02:03.599824 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa7885a2_c0c8_46f5_88c5_911bf0f9e9c5.slice/crio-68c0218fe854397e00e83e32f203425ff92dfa43a76312e2dc5e0d488fec20fb WatchSource:0}: Error finding container 68c0218fe854397e00e83e32f203425ff92dfa43a76312e2dc5e0d488fec20fb: Status 404 returned error can't find the container with id 68c0218fe854397e00e83e32f203425ff92dfa43a76312e2dc5e0d488fec20fb Sep 09 16:02:04.589846 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:04.589586 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" event={"ID":"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5","Type":"ContainerStarted","Data":"68c0218fe854397e00e83e32f203425ff92dfa43a76312e2dc5e0d488fec20fb"} Sep 09 16:02:08.559100 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.559073 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-w4nn8" Sep 09 16:02:08.616920 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.616589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" event={"ID":"fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5","Type":"ContainerStarted","Data":"3ea0ad71386c3d51322f2e19563460474859825293930c4008768da4201567d0"} Sep 09 16:02:08.617088 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.616962 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:08.618620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.618594 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" event={"ID":"3cdc679a-6c26-406b-8223-49f2a4c44b06","Type":"ContainerStarted","Data":"38205554b7b5d5d144b128205317372968c1b776c3f7647e88ad01d6e36ba2c7"} Sep 09 16:02:08.618736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.618644 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:02:08.635711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.635662 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" podStartSLOduration=2.62087106 podStartE2EDuration="6.635644752s" totalImagesPullingTime="4.014773692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:08.632984578 +0000 UTC m=+647.974176407" watchObservedRunningTime="2026-09-09 16:02:08.635644752 +0000 UTC m=+647.976836562" Sep 09 16:02:08.652115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:08.652076 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" podStartSLOduration=2.654445701 podStartE2EDuration="6.652059832s" totalImagesPullingTime="3.997614131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:08.651061367 +0000 UTC m=+647.992253196" watchObservedRunningTime="2026-09-09 16:02:08.652059832 +0000 UTC m=+647.993251659" Sep 09 16:02:09.624876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:09.624831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" event={"ID":"2b7067f9-9854-4b29-81cf-d6f41c4da5d4","Type":"ContainerStarted","Data":"e09bed80397c8af52a10e53e2747b1ff0fedfac53e4938f5a2ce00d513af45eb"} Sep 09 16:02:09.625273 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:09.625118 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:09.644476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:09.644432 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" podStartSLOduration=1.5370317070000001 podStartE2EDuration="9.644404247s" totalImagesPullingTime="8.10737254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:09.642822943 +0000 UTC m=+648.984014771" watchObservedRunningTime="2026-09-09 16:02:09.644404247 +0000 UTC m=+648.985596074" Sep 09 16:02:20.637529 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:20.637496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-678f94cf5f-h886r" Sep 09 16:02:39.627019 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:39.626985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zmlrh" Sep 09 16:02:39.630509 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:02:39.630484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-scjp2" Sep 09 16:03:31.498995 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.498916 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv"] Sep 09 16:03:31.501324 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.501306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:31.503930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.503905 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 16:03:31.504060 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.503904 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 16:03:31.504060 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.503905 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 16:03:31.504060 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.503966 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g5phn\"" Sep 09 16:03:31.510879 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.510852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv"] Sep 09 16:03:31.608356 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.608315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbz2\" (UniqueName: \"kubernetes.io/projected/cfdb095b-8755-4e8b-8ba1-5b63c7961fb5-kube-api-access-7qbz2\") pod \"dns-operator-controller-manager-7b84cd7c9c-9wwgv\" (UID: \"cfdb095b-8755-4e8b-8ba1-5b63c7961fb5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:31.709054 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.709017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbz2\" (UniqueName: \"kubernetes.io/projected/cfdb095b-8755-4e8b-8ba1-5b63c7961fb5-kube-api-access-7qbz2\") pod \"dns-operator-controller-manager-7b84cd7c9c-9wwgv\" (UID: \"cfdb095b-8755-4e8b-8ba1-5b63c7961fb5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:31.729817 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.729786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbz2\" (UniqueName: \"kubernetes.io/projected/cfdb095b-8755-4e8b-8ba1-5b63c7961fb5-kube-api-access-7qbz2\") pod \"dns-operator-controller-manager-7b84cd7c9c-9wwgv\" (UID: \"cfdb095b-8755-4e8b-8ba1-5b63c7961fb5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:31.811865 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:31.811774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:32.083643 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:32.083619 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv"] Sep 09 16:03:32.089166 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:03:32.089134 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdb095b_8755_4e8b_8ba1_5b63c7961fb5.slice/crio-c47d4c29545bdf9fa531656785b54049a6a1e14d40a0d40946c38ede2022f39b WatchSource:0}: Error finding container c47d4c29545bdf9fa531656785b54049a6a1e14d40a0d40946c38ede2022f39b: Status 404 returned error can't find the container with id c47d4c29545bdf9fa531656785b54049a6a1e14d40a0d40946c38ede2022f39b Sep 09 16:03:33.025004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:33.024971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" event={"ID":"cfdb095b-8755-4e8b-8ba1-5b63c7961fb5","Type":"ContainerStarted","Data":"c47d4c29545bdf9fa531656785b54049a6a1e14d40a0d40946c38ede2022f39b"} Sep 09 16:03:35.037114 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:35.037063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" event={"ID":"cfdb095b-8755-4e8b-8ba1-5b63c7961fb5","Type":"ContainerStarted","Data":"88c9e18bbb22ecdd42292394a16f475762f64f37d9773ac9435283a17f47c1a4"} Sep 09 16:03:35.037595 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:35.037154 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:35.069037 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:35.068980 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" podStartSLOduration=1.205887863 podStartE2EDuration="4.068963107s" totalImagesPullingTime="2.863075244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:03:35.067596475 +0000 UTC m=+734.408788303" watchObservedRunningTime="2026-09-09 16:03:35.068963107 +0000 UTC m=+734.410154935" Sep 09 16:03:38.697076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.697041 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bfgph"] Sep 09 16:03:38.699593 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.699576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:38.702260 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.702238 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rtw75\"" Sep 09 16:03:38.711548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.711526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bfgph"] Sep 09 16:03:38.785039 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.785003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzg6\" (UniqueName: \"kubernetes.io/projected/6695ba1f-eae7-41cd-a52e-ca12927fe568-kube-api-access-ffzg6\") pod \"authorino-operator-76b478bd78-bfgph\" (UID: \"6695ba1f-eae7-41cd-a52e-ca12927fe568\") " pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:38.885801 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.885762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzg6\" (UniqueName: \"kubernetes.io/projected/6695ba1f-eae7-41cd-a52e-ca12927fe568-kube-api-access-ffzg6\") pod \"authorino-operator-76b478bd78-bfgph\" (UID: \"6695ba1f-eae7-41cd-a52e-ca12927fe568\") " pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:38.894328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:38.894303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzg6\" (UniqueName: \"kubernetes.io/projected/6695ba1f-eae7-41cd-a52e-ca12927fe568-kube-api-access-ffzg6\") pod \"authorino-operator-76b478bd78-bfgph\" (UID: \"6695ba1f-eae7-41cd-a52e-ca12927fe568\") " pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:39.009596 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:39.009506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:39.288811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:39.288725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bfgph"] Sep 09 16:03:39.292153 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:03:39.292121 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6695ba1f_eae7_41cd_a52e_ca12927fe568.slice/crio-3dd4612b7cea5558bc9d399099205acce69704bf6acf5dc5299e71d150016c2e WatchSource:0}: Error finding container 3dd4612b7cea5558bc9d399099205acce69704bf6acf5dc5299e71d150016c2e: Status 404 returned error can't find the container with id 3dd4612b7cea5558bc9d399099205acce69704bf6acf5dc5299e71d150016c2e Sep 09 16:03:40.063204 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:40.063166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" event={"ID":"6695ba1f-eae7-41cd-a52e-ca12927fe568","Type":"ContainerStarted","Data":"3dd4612b7cea5558bc9d399099205acce69704bf6acf5dc5299e71d150016c2e"} Sep 09 16:03:41.078245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:41.078159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" event={"ID":"6695ba1f-eae7-41cd-a52e-ca12927fe568","Type":"ContainerStarted","Data":"7e268a7a692d1ae8dad1dfb5ce1f9acd371c552cac2573baed85ae5d683a6cdf"} Sep 09 16:03:41.078635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:41.078259 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:41.094946 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:41.094889 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" podStartSLOduration=1.639553815 podStartE2EDuration="3.094872624s" totalImagesPullingTime="1.455318809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:03:41.093913317 +0000 UTC m=+740.435105143" watchObservedRunningTime="2026-09-09 16:03:41.094872624 +0000 UTC m=+740.436064453" Sep 09 16:03:46.044324 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:46.044292 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-9wwgv" Sep 09 16:03:52.084822 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:52.084792 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bfgph" Sep 09 16:03:53.427736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.427692 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq"] Sep 09 16:03:53.430054 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.430037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.432716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.432694 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lgrv5\"" Sep 09 16:03:53.442880 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.442852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq"] Sep 09 16:03:53.521483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.521448 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq"] Sep 09 16:03:53.521719 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:03:53.521699 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-gh2v4], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-gh2v4]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" podUID="f872b9bc-7045-47f8-b886-b4874b632390" Sep 09 16:03:53.621348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.621310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2v4\" (UniqueName: \"kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.621547 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.621366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.722022 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.721938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.722159 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.722059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2v4\" (UniqueName: \"kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.722328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.722308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:53.733695 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:53.733673 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2v4\" (UniqueName: \"kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:54.146686 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.146654 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:54.152874 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.152853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:54.327139 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.327101 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume\") pod \"f872b9bc-7045-47f8-b886-b4874b632390\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " Sep 09 16:03:54.327319 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.327249 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh2v4\" (UniqueName: \"kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4\") pod \"f872b9bc-7045-47f8-b886-b4874b632390\" (UID: \"f872b9bc-7045-47f8-b886-b4874b632390\") " Sep 09 16:03:54.327486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.327465 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f872b9bc-7045-47f8-b886-b4874b632390" (UID: "f872b9bc-7045-47f8-b886-b4874b632390"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:03:54.327579 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.327565 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f872b9bc-7045-47f8-b886-b4874b632390-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:03:54.329403 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.329374 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4" (OuterVolumeSpecName: "kube-api-access-gh2v4") pod "f872b9bc-7045-47f8-b886-b4874b632390" (UID: "f872b9bc-7045-47f8-b886-b4874b632390"). InnerVolumeSpecName "kube-api-access-gh2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:03:54.372323 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.372284 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq"] Sep 09 16:03:54.378440 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.378395 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq"] Sep 09 16:03:54.423242 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.423162 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:03:54.426215 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.426189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.428817 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.428789 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2v4\" (UniqueName: \"kubernetes.io/projected/f872b9bc-7045-47f8-b886-b4874b632390-kube-api-access-gh2v4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:03:54.459023 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.458980 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:03:54.529514 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.529404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5mt\" (UniqueName: \"kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.529686 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.529591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.631101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.631065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5mt\" (UniqueName: \"kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.631336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.631152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.631572 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.631554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.643570 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.643544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5mt\" (UniqueName: \"kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fxjdn\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:54.738130 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:54.738039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:55.014581 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:55.014557 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:03:55.016935 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:03:55.016905 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d6eb58_4302_4188_8157_641228d72707.slice/crio-eaaa6280115f80ed8868c35eefa1181d092b6b5bcaefdafdd9a7a96ffb13ae58 WatchSource:0}: Error finding container eaaa6280115f80ed8868c35eefa1181d092b6b5bcaefdafdd9a7a96ffb13ae58: Status 404 returned error can't find the container with id eaaa6280115f80ed8868c35eefa1181d092b6b5bcaefdafdd9a7a96ffb13ae58 Sep 09 16:03:55.153043 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:55.153005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" event={"ID":"81d6eb58-4302-4188-8157-641228d72707","Type":"ContainerStarted","Data":"eaaa6280115f80ed8868c35eefa1181d092b6b5bcaefdafdd9a7a96ffb13ae58"} Sep 09 16:03:55.153225 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:55.153057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:55.155448 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:03:55.155403 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="f872b9bc-7045-47f8-b886-b4874b632390" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:55.161603 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:03:55.161573 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s9sgq\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="f872b9bc-7045-47f8-b886-b4874b632390" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s9sgq" Sep 09 16:03:55.190124 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:55.190090 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f872b9bc-7045-47f8-b886-b4874b632390" path="/var/lib/kubelet/pods/f872b9bc-7045-47f8-b886-b4874b632390/volumes" Sep 09 16:03:59.177386 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:59.177352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" event={"ID":"81d6eb58-4302-4188-8157-641228d72707","Type":"ContainerStarted","Data":"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634"} Sep 09 16:03:59.177769 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:59.177463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:03:59.203116 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:03:59.203064 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" podStartSLOduration=1.684211693 podStartE2EDuration="5.203049098s" totalImagesPullingTime="3.518837405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:03:59.202210626 +0000 UTC m=+758.543402459" watchObservedRunningTime="2026-09-09 16:03:59.203049098 +0000 UTC m=+758.544240927" Sep 09 16:04:06.359696 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.359657 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:04:06.360202 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.359891 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" podUID="81d6eb58-4302-4188-8157-641228d72707" containerName="manager" containerID="cri-o://c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634" gracePeriod=10 Sep 09 16:04:06.361829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.361800 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:04:06.610316 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.610255 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:04:06.639325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.639294 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume\") pod \"81d6eb58-4302-4188-8157-641228d72707\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " Sep 09 16:04:06.639492 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.639387 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd5mt\" (UniqueName: \"kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt\") pod \"81d6eb58-4302-4188-8157-641228d72707\" (UID: \"81d6eb58-4302-4188-8157-641228d72707\") " Sep 09 16:04:06.639854 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.639833 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "81d6eb58-4302-4188-8157-641228d72707" (UID: "81d6eb58-4302-4188-8157-641228d72707"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:04:06.641679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.641652 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt" (OuterVolumeSpecName: "kube-api-access-pd5mt") pod "81d6eb58-4302-4188-8157-641228d72707" (UID: "81d6eb58-4302-4188-8157-641228d72707"). InnerVolumeSpecName "kube-api-access-pd5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:04:06.741007 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.740972 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81d6eb58-4302-4188-8157-641228d72707-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:04:06.741007 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:06.741003 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5mt\" (UniqueName: \"kubernetes.io/projected/81d6eb58-4302-4188-8157-641228d72707-kube-api-access-pd5mt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:04:07.218069 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.218038 2562 generic.go:356] "Generic (PLEG): container finished" podID="81d6eb58-4302-4188-8157-641228d72707" containerID="c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634" exitCode=0 Sep 09 16:04:07.218208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.218082 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" Sep 09 16:04:07.218208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.218110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" event={"ID":"81d6eb58-4302-4188-8157-641228d72707","Type":"ContainerDied","Data":"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634"} Sep 09 16:04:07.218208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.218149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn" event={"ID":"81d6eb58-4302-4188-8157-641228d72707","Type":"ContainerDied","Data":"eaaa6280115f80ed8868c35eefa1181d092b6b5bcaefdafdd9a7a96ffb13ae58"} Sep 09 16:04:07.218208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.218170 2562 scope.go:117] "RemoveContainer" containerID="c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634" Sep 09 16:04:07.227076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.227058 2562 scope.go:117] "RemoveContainer" containerID="c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634" Sep 09 16:04:07.227299 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:04:07.227281 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634\": container with ID starting with c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634 not found: ID does not exist" containerID="c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634" Sep 09 16:04:07.227342 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.227305 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634"} err="failed to get container status \"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634\": rpc error: code = NotFound desc = could not find container \"c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634\": container with ID starting with c091137397d5e0148518ab6ebb5e60672d424d403e9b935fc205e15f48c0b634 not found: ID does not exist" Sep 09 16:04:07.240529 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.240508 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:04:07.244697 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:07.244677 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fxjdn"] Sep 09 16:04:09.196484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:09.196451 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d6eb58-4302-4188-8157-641228d72707" path="/var/lib/kubelet/pods/81d6eb58-4302-4188-8157-641228d72707/volumes" Sep 09 16:04:22.751248 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.751214 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h"] Sep 09 16:04:22.751757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.751733 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d6eb58-4302-4188-8157-641228d72707" containerName="manager" Sep 09 16:04:22.751802 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.751760 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d6eb58-4302-4188-8157-641228d72707" containerName="manager" Sep 09 16:04:22.751895 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.751883 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d6eb58-4302-4188-8157-641228d72707" containerName="manager" Sep 09 16:04:22.756164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.756144 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.758799 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.758776 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4v6vv\"" Sep 09 16:04:22.765198 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.765173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h"] Sep 09 16:04:22.888187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888389 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888273 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888389 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888389 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888509 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888509 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsg4\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-kube-api-access-5dsg4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888509 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.888613 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.888521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsg4\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-kube-api-access-5dsg4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989916 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.989967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.990078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.989981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.990286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.990245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.990436 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.990331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.990436 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.990359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.990626 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.990607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.992164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.992132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.992291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.992274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.998779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.998751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:22.999187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:22.999165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsg4\" (UniqueName: \"kubernetes.io/projected/13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3-kube-api-access-5dsg4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vrq8h\" (UID: \"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:23.068004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:23.067919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:23.341722 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:23.341693 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h"] Sep 09 16:04:23.344116 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:04:23.344088 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b41dac_ff54_4a7a_bd94_0dc9e85bf1d3.slice/crio-eff161eddbc23935e8f1b6ea6ea3f2660746c1b5de89fd527b66d7713c877590 WatchSource:0}: Error finding container eff161eddbc23935e8f1b6ea6ea3f2660746c1b5de89fd527b66d7713c877590: Status 404 returned error can't find the container with id eff161eddbc23935e8f1b6ea6ea3f2660746c1b5de89fd527b66d7713c877590 Sep 09 16:04:23.349532 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:23.349489 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:04:23.349644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:23.349581 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:04:23.349644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:23.349631 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:04:24.302828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:24.302797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" event={"ID":"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3","Type":"ContainerStarted","Data":"a7084951214898070190188e7ac9a1aa8f12c61ab230acd73bafce4d986ed1d6"} Sep 09 16:04:24.303209 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:24.302854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" event={"ID":"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3","Type":"ContainerStarted","Data":"eff161eddbc23935e8f1b6ea6ea3f2660746c1b5de89fd527b66d7713c877590"} Sep 09 16:04:24.323559 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:24.323506 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podStartSLOduration=2.323490717 podStartE2EDuration="2.323490717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:04:24.321230852 +0000 UTC m=+783.662422683" watchObservedRunningTime="2026-09-09 16:04:24.323490717 +0000 UTC m=+783.664682545" Sep 09 16:04:25.069055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:25.069015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:25.073912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:25.073885 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:25.307362 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:25.307325 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:25.308333 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:25.308312 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:04:36.570382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.570347 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:04:36.572887 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.572864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.575385 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.575359 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 16:04:36.575513 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.575474 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mfjz\"" Sep 09 16:04:36.582772 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.582746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:04:36.606255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.606232 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:04:36.712526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.712489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b047e1-f0a3-48d0-be95-a509d962b640-config-file\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.712712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.712615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/43b047e1-f0a3-48d0-be95-a509d962b640-kube-api-access-sk64s\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.813731 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.813692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b047e1-f0a3-48d0-be95-a509d962b640-config-file\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.813964 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.813802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/43b047e1-f0a3-48d0-be95-a509d962b640-kube-api-access-sk64s\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.814460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.814439 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/43b047e1-f0a3-48d0-be95-a509d962b640-config-file\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.823011 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.822942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk64s\" (UniqueName: \"kubernetes.io/projected/43b047e1-f0a3-48d0-be95-a509d962b640-kube-api-access-sk64s\") pod \"limitador-limitador-84994f7ffc-8tlvm\" (UID: \"43b047e1-f0a3-48d0-be95-a509d962b640\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:36.884904 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:36.884871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:37.157317 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:37.157293 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:04:37.159180 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:04:37.159141 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b047e1_f0a3_48d0_be95_a509d962b640.slice/crio-5677b7d231c3fc7e29dd94bb60dc21970ad2e73165e3f88bbb47fd889746abee WatchSource:0}: Error finding container 5677b7d231c3fc7e29dd94bb60dc21970ad2e73165e3f88bbb47fd889746abee: Status 404 returned error can't find the container with id 5677b7d231c3fc7e29dd94bb60dc21970ad2e73165e3f88bbb47fd889746abee Sep 09 16:04:37.368348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:37.368316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" event={"ID":"43b047e1-f0a3-48d0-be95-a509d962b640","Type":"ContainerStarted","Data":"5677b7d231c3fc7e29dd94bb60dc21970ad2e73165e3f88bbb47fd889746abee"} Sep 09 16:04:38.374773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:38.374734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" event={"ID":"43b047e1-f0a3-48d0-be95-a509d962b640","Type":"ContainerStarted","Data":"0404152a8e6f4d4518b71ddba1b8eed728596f5d3def184bf46c01b1abdf4edd"} Sep 09 16:04:38.375155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:38.374831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:04:38.391349 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:38.391300 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" podStartSLOduration=1.308086582 podStartE2EDuration="2.39128708s" totalImagesPullingTime="1.083200498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:04:38.390191591 +0000 UTC m=+797.731383431" watchObservedRunningTime="2026-09-09 16:04:38.39128708 +0000 UTC m=+797.732478943" Sep 09 16:04:49.381707 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:04:49.381674 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8tlvm" Sep 09 16:05:51.826639 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:05:51.826604 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:06:01.831969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.831933 2562 generic.go:356] "Generic (PLEG): container finished" podID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerID="a7084951214898070190188e7ac9a1aa8f12c61ab230acd73bafce4d986ed1d6" exitCode=0 Sep 09 16:06:01.831969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.831973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" event={"ID":"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3","Type":"ContainerDied","Data":"a7084951214898070190188e7ac9a1aa8f12c61ab230acd73bafce4d986ed1d6"} Sep 09 16:06:01.832400 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.832339 2562 scope.go:117] "RemoveContainer" containerID="a7084951214898070190188e7ac9a1aa8f12c61ab230acd73bafce4d986ed1d6" Sep 09 16:06:01.833373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.833340 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:06:01.833460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.833427 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:06:01.833498 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:01.833468 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:06:02.839230 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:02.839194 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" event={"ID":"13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3","Type":"ContainerStarted","Data":"d00e2a53eb59e18a31e27373ed82ae3d3bc13c46ddf955e399944089dc54c741"} Sep 09 16:06:03.068912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:03.068878 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:06:03.069107 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:03.069089 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vrq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 09 16:06:03.069183 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:03.069146 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podUID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 09 16:06:04.068399 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:04.068360 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vrq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 09 16:06:04.068803 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:04.068441 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podUID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 09 16:06:05.068666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:05.068635 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vrq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 09 16:06:05.069061 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:05.068687 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podUID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 09 16:06:06.068486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:06.068447 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vrq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 09 16:06:06.068663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:06.068513 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podUID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 09 16:06:07.068576 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:07.068535 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vrq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 09 16:06:07.068972 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:07.068602 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" podUID="13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 09 16:06:08.072752 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:08.072717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:06:08.073182 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:08.072968 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:06:08.073736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:08.073719 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vrq8h" Sep 09 16:06:16.526332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:16.526251 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:06:23.643813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:23.643783 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:06:23.644439 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:23.644376 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:06:27.247037 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:27.247000 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:06:34.825532 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:06:34.825494 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:07:03.520549 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:03.520517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:07:07.832973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.832935 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk"] Sep 09 16:07:07.835395 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.835374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.839347 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.839318 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 16:07:07.839500 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.839452 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kzfjj\"" Sep 09 16:07:07.839569 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.839559 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 16:07:07.839667 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.839650 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 09 16:07:07.845259 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.845237 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk"] Sep 09 16:07:07.878526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27aa575-0cdc-405e-bb53-ec7e71869820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.878526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.878715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.878715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9t44\" (UniqueName: \"kubernetes.io/projected/c27aa575-0cdc-405e-bb53-ec7e71869820-kube-api-access-f9t44\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.878715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.878715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.878684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27aa575-0cdc-405e-bb53-ec7e71869820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979292 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979292 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t44\" (UniqueName: \"kubernetes.io/projected/c27aa575-0cdc-405e-bb53-ec7e71869820-kube-api-access-f9t44\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979292 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979292 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979536 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.979842 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.979814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.981529 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.981506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27aa575-0cdc-405e-bb53-ec7e71869820-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.982015 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.981994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27aa575-0cdc-405e-bb53-ec7e71869820-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:07.987155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:07.987131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t44\" (UniqueName: \"kubernetes.io/projected/c27aa575-0cdc-405e-bb53-ec7e71869820-kube-api-access-f9t44\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk\" (UID: \"c27aa575-0cdc-405e-bb53-ec7e71869820\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:08.147371 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:08.147340 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:08.418453 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:08.418420 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk"] Sep 09 16:07:08.420004 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:07:08.419980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27aa575_0cdc_405e_bb53_ec7e71869820.slice/crio-f7121298f62d94d3abe3c2951509b9814ccb673f32b44901c6bfef31c54e1461 WatchSource:0}: Error finding container f7121298f62d94d3abe3c2951509b9814ccb673f32b44901c6bfef31c54e1461: Status 404 returned error can't find the container with id f7121298f62d94d3abe3c2951509b9814ccb673f32b44901c6bfef31c54e1461 Sep 09 16:07:08.421474 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:08.421457 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 16:07:09.196540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:09.196501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" event={"ID":"c27aa575-0cdc-405e-bb53-ec7e71869820","Type":"ContainerStarted","Data":"f7121298f62d94d3abe3c2951509b9814ccb673f32b44901c6bfef31c54e1461"} Sep 09 16:07:09.324822 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:09.324324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:07:10.207982 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:10.207941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" event={"ID":"c27aa575-0cdc-405e-bb53-ec7e71869820","Type":"ContainerStarted","Data":"cd5eaefd7a8f9c57beb5b4ee27dee00c0acbdc31ccec48d17b60058fd986c768"} Sep 09 16:07:10.225019 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:10.224968 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" podStartSLOduration=2.246917658 podStartE2EDuration="3.224950083s" totalImagesPullingTime="978.032425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:07:10.223772411 +0000 UTC m=+949.564964242" watchObservedRunningTime="2026-09-09 16:07:10.224950083 +0000 UTC m=+949.566141911" Sep 09 16:07:18.147630 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:18.147591 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:18.148074 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:18.147650 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:18.159916 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:18.159890 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:18.259907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:18.259882 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk" Sep 09 16:07:18.735890 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:18.735859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:07:21.333199 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.333167 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw"] Sep 09 16:07:21.336975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.336951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.339507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.339484 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 09 16:07:21.348184 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.348161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw"] Sep 09 16:07:21.388463 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.388633 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.388633 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b150a0a3-f378-4f60-ae27-00cef5212f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.388633 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.388795 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.388795 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.388706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q4xf\" (UniqueName: \"kubernetes.io/projected/b150a0a3-f378-4f60-ae27-00cef5212f2c-kube-api-access-4q4xf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.489922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.489886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b150a0a3-f378-4f60-ae27-00cef5212f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.489929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.489973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.489994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q4xf\" (UniqueName: \"kubernetes.io/projected/b150a0a3-f378-4f60-ae27-00cef5212f2c-kube-api-access-4q4xf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.490067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490280 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.490101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490518 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.490495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.490571 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.490687 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.490645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.492215 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.492187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b150a0a3-f378-4f60-ae27-00cef5212f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.492384 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.492365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b150a0a3-f378-4f60-ae27-00cef5212f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.497618 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.497598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q4xf\" (UniqueName: \"kubernetes.io/projected/b150a0a3-f378-4f60-ae27-00cef5212f2c-kube-api-access-4q4xf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw\" (UID: \"b150a0a3-f378-4f60-ae27-00cef5212f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.646496 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.646459 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:21.922212 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:21.922187 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw"] Sep 09 16:07:21.924210 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:07:21.924184 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb150a0a3_f378_4f60_ae27_00cef5212f2c.slice/crio-f70893c9fab3c023d25b4300d1f3b958786bf4dc0e9d82b77ee174e605754354 WatchSource:0}: Error finding container f70893c9fab3c023d25b4300d1f3b958786bf4dc0e9d82b77ee174e605754354: Status 404 returned error can't find the container with id f70893c9fab3c023d25b4300d1f3b958786bf4dc0e9d82b77ee174e605754354 Sep 09 16:07:22.268683 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:22.268652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" event={"ID":"b150a0a3-f378-4f60-ae27-00cef5212f2c","Type":"ContainerStarted","Data":"5fcb9657c32895755a8860a40f1a6de97742fdf1f1844bbe3173f3b253080dcf"} Sep 09 16:07:22.268829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:22.268690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" event={"ID":"b150a0a3-f378-4f60-ae27-00cef5212f2c","Type":"ContainerStarted","Data":"f70893c9fab3c023d25b4300d1f3b958786bf4dc0e9d82b77ee174e605754354"} Sep 09 16:07:22.288396 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:22.288332 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" podStartSLOduration=1.070797967 podStartE2EDuration="1.288312191s" totalImagesPullingTime="217.514224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:07:22.283785075 +0000 UTC m=+961.624976925" watchObservedRunningTime="2026-09-09 16:07:22.288312191 +0000 UTC m=+961.629504022" Sep 09 16:07:27.426225 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:27.426190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:07:31.647345 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:31.647305 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:31.647753 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:31.647388 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:31.659847 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:31.659824 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:07:32.330580 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:07:32.330547 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw" Sep 09 16:09:08.133433 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.133365 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:09:08.137483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.137449 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.139882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.139860 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7b8369-openshift-default-dockercfg-zbkfs\"" Sep 09 16:09:08.149697 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.149667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:09:08.290459 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290634 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290634 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290725 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290772 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290845 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nch5n\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.290972 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.291012 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.290993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.391785 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.391785 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.391785 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nch5n\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.391979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392339 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.392172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392339 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.392197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.392459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392550 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.392495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.392712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.392694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.394232 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.394211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.394531 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.394511 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.399674 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.399649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.399833 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.399818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nch5n\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n\") pod \"e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.450192 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.450157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:08.730070 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.730040 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:09:08.732128 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:08.732098 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd744b4_2594_4359_bff6_07aa0c8d5bec.slice/crio-0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049 WatchSource:0}: Error finding container 0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049: Status 404 returned error can't find the container with id 0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049 Sep 09 16:09:08.734497 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.734460 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:08.734594 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.734554 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:08.734657 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.734601 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:08.831152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.831114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerStarted","Data":"9de04cccec4ad46f2e163322e1693128030ede13ede4c821442881cb5f0816fc"} Sep 09 16:09:08.831344 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.831180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerStarted","Data":"0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049"} Sep 09 16:09:08.852036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:08.851981 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podStartSLOduration=0.85196505 podStartE2EDuration="851.96505ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:08.848798906 +0000 UTC m=+1068.189990738" watchObservedRunningTime="2026-09-09 16:09:08.85196505 +0000 UTC m=+1068.193156879" Sep 09 16:09:09.450462 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:09.450402 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:09.455364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:09.455337 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:09.837393 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:09.837308 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:09.838246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:09.838226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:11.325169 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:11.325131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:09:16.319350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.319260 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:09:16.323864 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.323839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.326751 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.326728 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-083463c6-openshift-default-dockercfg-58d4d\"" Sep 09 16:09:16.334033 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.334009 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:09:16.474505 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474671 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474671 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474750 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474750 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474750 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474750 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxrf\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.474903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.474828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.575743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.575743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.575952 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.575952 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.575952 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.575989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxrf\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576295 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576295 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576295 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576481 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.576682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.576656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.578161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.578140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.578365 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.578347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.584922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.584897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.585105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.585089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxrf\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf\") pod \"e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.635681 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.635655 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:16.913832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.913801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:09:16.915762 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:16.915738 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014a766d_98a5_4f8a_8d1d_5d8cf2b2c218.slice/crio-09fb09986d68d241084986b1b42dcff7201826a60a6f292779cda7fda4e3e1f5 WatchSource:0}: Error finding container 09fb09986d68d241084986b1b42dcff7201826a60a6f292779cda7fda4e3e1f5: Status 404 returned error can't find the container with id 09fb09986d68d241084986b1b42dcff7201826a60a6f292779cda7fda4e3e1f5 Sep 09 16:09:16.918029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.917995 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:16.918118 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.918060 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:16.918157 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:16.918115 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:17.067371 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.067080 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:09:17.072044 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.072021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.074756 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.074731 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a09c2a-openshift-default-dockercfg-8fhbr\"" Sep 09 16:09:17.083460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.083440 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:09:17.180912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.180811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181169 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcms\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181520 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181647 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181761 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.181867 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.181854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.182019 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.182006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.182134 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.182121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.283460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.283400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcms\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.283735 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.283713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.283885 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.283870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.283998 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.283982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.284116 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.284102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.284231 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.284218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.284715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.284690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285425 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285625 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285753 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.284838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.285936 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.286015 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.285995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.288482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.288459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.290039 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.289108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.292576 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.292532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcms\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.296430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.293239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token\") pod \"e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.389376 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.389276 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:17.677816 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.677785 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:09:17.680779 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:17.679784 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1935ec5f_226d_4eb7_9292_6343c8c69da7.slice/crio-24f5c9305b2108095ba740b0a4abdffb2806cdd632365b59dfdc08d79d761a0d WatchSource:0}: Error finding container 24f5c9305b2108095ba740b0a4abdffb2806cdd632365b59dfdc08d79d761a0d: Status 404 returned error can't find the container with id 24f5c9305b2108095ba740b0a4abdffb2806cdd632365b59dfdc08d79d761a0d Sep 09 16:09:17.683997 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.683950 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:17.684087 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.684050 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:17.684152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.684091 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:17.880085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.879327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerStarted","Data":"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db"} Sep 09 16:09:17.880085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.879380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerStarted","Data":"24f5c9305b2108095ba740b0a4abdffb2806cdd632365b59dfdc08d79d761a0d"} Sep 09 16:09:17.884050 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.884014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerStarted","Data":"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7"} Sep 09 16:09:17.884189 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.884091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerStarted","Data":"09fb09986d68d241084986b1b42dcff7201826a60a6f292779cda7fda4e3e1f5"} Sep 09 16:09:17.918119 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.918057 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podStartSLOduration=1.918036411 podStartE2EDuration="1.918036411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:17.914952765 +0000 UTC m=+1077.256144621" watchObservedRunningTime="2026-09-09 16:09:17.918036411 +0000 UTC m=+1077.259228241" Sep 09 16:09:17.919479 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:17.919433 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podStartSLOduration=0.919397286 podStartE2EDuration="919.397286ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:17.899706321 +0000 UTC m=+1077.240898167" watchObservedRunningTime="2026-09-09 16:09:17.919397286 +0000 UTC m=+1077.260589117" Sep 09 16:09:18.390020 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.389981 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:18.391932 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.391896 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:18.392068 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.391956 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:18.636815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.636786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:18.641656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.641581 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:18.895155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.895057 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:18.903105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:18.903069 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:19.399915 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:19.399679 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:19.909574 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:19.909400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:19.911226 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:19.911202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:21.040398 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:21.040362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:09:25.631830 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.631374 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": read tcp 10.132.0.2:55680->10.132.0.40:15021: read: connection reset by peer" start-of-body= Sep 09 16:09:25.631830 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.631533 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": read tcp 10.132.0.2:55680->10.132.0.40:15021: read: connection reset by peer" Sep 09 16:09:25.943055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.942969 2562 generic.go:356] "Generic (PLEG): container finished" podID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerID="9de04cccec4ad46f2e163322e1693128030ede13ede4c821442881cb5f0816fc" exitCode=0 Sep 09 16:09:25.943055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.943023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerDied","Data":"9de04cccec4ad46f2e163322e1693128030ede13ede4c821442881cb5f0816fc"} Sep 09 16:09:25.943483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.943469 2562 scope.go:117] "RemoveContainer" containerID="9de04cccec4ad46f2e163322e1693128030ede13ede4c821442881cb5f0816fc" Sep 09 16:09:25.944553 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.944527 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:25.944630 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.944592 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:25.944674 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:25.944630 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:26.949873 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:26.949835 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerStarted","Data":"f27864883b0c271883c4a0288f8a1a448766682bffcc040ddd5b8f7df9820440"} Sep 09 16:09:27.451326 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:27.451287 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:27.451519 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:27.451479 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:27.451572 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:27.451525 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:28.450903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:28.450864 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:28.451289 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:28.450923 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:29.451608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:29.451115 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:29.451608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:29.451183 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:30.451118 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:30.451075 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:30.451399 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:30.451367 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:31.450673 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:31.450638 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:31.451663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:31.451629 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:32.451029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:32.450976 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 09 16:09:32.451557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:32.451055 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 09 16:09:33.128641 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:33.128602 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:09:33.454421 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:33.454315 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:33.454838 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:33.454701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:33.455349 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:33.455321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:09:34.936625 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:34.936566 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:09:34.937070 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:34.936639 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:09:35.951650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:35.951595 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:09:35.952155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:35.951675 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:09:36.546825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.546788 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:09:36.552050 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.552020 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.554789 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.554762 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c23ff1-openshift-default-dockercfg-rt44n\"" Sep 09 16:09:36.565550 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.565521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:09:36.637631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.637631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.637631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.637980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.638023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.638328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.638098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5mq\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739156 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5mq\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.739679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.739562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.740059 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.740011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.740059 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.740028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.740246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.740133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.740246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.740153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.742031 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.742000 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.742145 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.742051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.747722 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.747698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5mq\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.747881 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.747860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token\") pod \"e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:36.870895 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:36.870860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:37.153290 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:37.153253 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:09:37.154674 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:37.154645 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c88c80_ba00_43c7_bb0b_93b5c1529399.slice/crio-55c06c5c1ecb512bd2bd53f35ad32afc77c502bb9131aa2e3ea1b93b2e9d757e WatchSource:0}: Error finding container 55c06c5c1ecb512bd2bd53f35ad32afc77c502bb9131aa2e3ea1b93b2e9d757e: Status 404 returned error can't find the container with id 55c06c5c1ecb512bd2bd53f35ad32afc77c502bb9131aa2e3ea1b93b2e9d757e Sep 09 16:09:37.157392 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:37.157351 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:37.157535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:37.157450 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:37.157535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:37.157499 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.020984 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.020869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerStarted","Data":"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324"} Sep 09 16:09:38.020984 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.020934 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerStarted","Data":"55c06c5c1ecb512bd2bd53f35ad32afc77c502bb9131aa2e3ea1b93b2e9d757e"} Sep 09 16:09:38.022572 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.022544 2562 generic.go:356] "Generic (PLEG): container finished" podID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerID="cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db" exitCode=0 Sep 09 16:09:38.022688 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.022608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerDied","Data":"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db"} Sep 09 16:09:38.023015 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.023000 2562 scope.go:117] "RemoveContainer" containerID="cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db" Sep 09 16:09:38.024168 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024131 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.024260 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024220 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.024322 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024269 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.024385 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024365 2562 generic.go:356] "Generic (PLEG): container finished" podID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerID="23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7" exitCode=0 Sep 09 16:09:38.024471 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerDied","Data":"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7"} Sep 09 16:09:38.024780 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.024765 2562 scope.go:117] "RemoveContainer" containerID="23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7" Sep 09 16:09:38.025825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.025795 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.025895 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.025867 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.025958 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.025906 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:09:38.043803 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.043765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podStartSLOduration=2.043751272 podStartE2EDuration="2.043751272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:38.042019043 +0000 UTC m=+1097.383210879" watchObservedRunningTime="2026-09-09 16:09:38.043751272 +0000 UTC m=+1097.384943100" Sep 09 16:09:38.871742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.871703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:38.876815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:38.876790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:39.034628 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.034593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerStarted","Data":"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080"} Sep 09 16:09:39.037593 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.037530 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerStarted","Data":"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621"} Sep 09 16:09:39.037692 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.037638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:39.038231 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.038211 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:09:39.390612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.390083 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:39.390612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.390199 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:39.390612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.390244 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:39.636385 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.636356 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:39.636629 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.636403 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 09 16:09:39.636756 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:39.636737 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 09 16:09:40.390553 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:40.390508 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:40.391043 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:40.390576 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:40.637010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:40.636974 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 09 16:09:40.637173 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:40.637040 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 09 16:09:41.391093 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:41.391052 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:41.391629 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:41.391121 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:41.636726 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:41.636682 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 09 16:09:41.636889 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:41.636763 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 09 16:09:42.390457 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:42.390359 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:42.390658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:42.390466 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:42.636761 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:42.636718 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 09 16:09:42.637262 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:42.636791 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 09 16:09:43.390316 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.390273 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:43.390607 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.390341 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:43.568736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.568698 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:09:43.573621 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.573601 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.576019 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.576003 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-049e4f89-kserve-self-signed-certs\"" Sep 09 16:09:43.586846 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.586821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:09:43.638475 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.637148 2562 patch_prober.go:28] interesting pod/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 09 16:09:43.638475 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.637223 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 09 16:09:43.651182 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.651182 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.651374 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.651374 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mmm\" (UniqueName: \"kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.651374 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.651579 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.651400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752441 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mmm\" (UniqueName: \"kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.752856 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.752666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.753283 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.753256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.753283 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.753271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.753557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.753537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.755453 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.755430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.755771 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.755752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.764814 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.764790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mmm\" (UniqueName: \"kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm\") pod \"e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:43.888805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:43.888704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:44.230182 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.230153 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:09:44.233100 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:44.233066 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99005dc3_a71e_4c41_8bfd_b8a4ba37933c.slice/crio-8b4ef8f2c00353d09dcf1a3965f7a10f5fe92c88e6f9e4fe548eced5f6b00e1c WatchSource:0}: Error finding container 8b4ef8f2c00353d09dcf1a3965f7a10f5fe92c88e6f9e4fe548eced5f6b00e1c: Status 404 returned error can't find the container with id 8b4ef8f2c00353d09dcf1a3965f7a10f5fe92c88e6f9e4fe548eced5f6b00e1c Sep 09 16:09:44.389609 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.389571 2562 patch_prober.go:28] interesting pod/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 09 16:09:44.389776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.389629 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 09 16:09:44.640506 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.640474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:44.640962 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.640855 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:44.641513 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.641493 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:09:44.863989 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.863958 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn"] Sep 09 16:09:44.867619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.867600 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:44.870543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.870524 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:09:44.870728 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.870715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:09:44.874501 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.874470 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn"] Sep 09 16:09:44.965873 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:44.965783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-083463c6-t7qnn\" (UID: \"da4b2cc1-f6af-4aec-8516-f08eb646c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:45.067366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.067323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-083463c6-t7qnn\" (UID: \"da4b2cc1-f6af-4aec-8516-f08eb646c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:45.067949 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.067928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-083463c6-t7qnn\" (UID: \"da4b2cc1-f6af-4aec-8516-f08eb646c9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:45.076297 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.076268 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" event={"ID":"99005dc3-a71e-4c41-8bfd-b8a4ba37933c","Type":"ContainerStarted","Data":"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324"} Sep 09 16:09:45.076388 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.076316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" event={"ID":"99005dc3-a71e-4c41-8bfd-b8a4ba37933c","Type":"ContainerStarted","Data":"8b4ef8f2c00353d09dcf1a3965f7a10f5fe92c88e6f9e4fe548eced5f6b00e1c"} Sep 09 16:09:45.094122 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.094078 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" podStartSLOduration=2.094061933 podStartE2EDuration="2.094061933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:09:45.092283314 +0000 UTC m=+1104.433475144" watchObservedRunningTime="2026-09-09 16:09:45.094061933 +0000 UTC m=+1104.435253756" Sep 09 16:09:45.179589 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.179556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:45.393461 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.393433 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:45.393754 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.393731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:45.394442 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.394423 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:09:45.466754 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:45.466730 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn"] Sep 09 16:09:45.469393 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:09:45.469355 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4b2cc1_f6af_4aec_8516_f08eb646c9a8.slice/crio-24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb WatchSource:0}: Error finding container 24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb: Status 404 returned error can't find the container with id 24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb Sep 09 16:09:46.083195 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:46.083149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" event={"ID":"da4b2cc1-f6af-4aec-8516-f08eb646c9a8","Type":"ContainerStarted","Data":"24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb"} Sep 09 16:09:49.028079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:49.027985 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:09:51.115123 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:51.115092 2562 generic.go:356] "Generic (PLEG): container finished" podID="da4b2cc1-f6af-4aec-8516-f08eb646c9a8" containerID="2d61a226095516c8fc060b014bf3f0382cfa0795cd4a0f050824cd9963b184f3" exitCode=0 Sep 09 16:09:51.115508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:51.115167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" event={"ID":"da4b2cc1-f6af-4aec-8516-f08eb646c9a8","Type":"ContainerDied","Data":"2d61a226095516c8fc060b014bf3f0382cfa0795cd4a0f050824cd9963b184f3"} Sep 09 16:09:52.274930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:52.274905 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:52.340004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:52.339967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca\") pod \"da4b2cc1-f6af-4aec-8516-f08eb646c9a8\" (UID: \"da4b2cc1-f6af-4aec-8516-f08eb646c9a8\") " Sep 09 16:09:52.340301 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:52.340277 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da4b2cc1-f6af-4aec-8516-f08eb646c9a8" (UID: "da4b2cc1-f6af-4aec-8516-f08eb646c9a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:09:52.340387 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:52.340363 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4b2cc1-f6af-4aec-8516-f08eb646c9a8-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:09:53.128631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.128603 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" Sep 09 16:09:53.128809 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.128634 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn" event={"ID":"da4b2cc1-f6af-4aec-8516-f08eb646c9a8","Type":"ContainerDied","Data":"24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb"} Sep 09 16:09:53.128809 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.128681 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24eb48e9f9078c2c878115f3dda5095de9131f7f56c01df831e59fd2188464eb" Sep 09 16:09:53.896218 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.896188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:53.896796 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.896776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:53.917010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:53.916986 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:09:54.145145 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:09:54.145118 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:10:01.170876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.170842 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn"] Sep 09 16:10:01.172844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.172823 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-083463c6-t7qnn"] Sep 09 16:10:01.184758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.184727 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerID="b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324" exitCode=0 Sep 09 16:10:01.190511 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.190488 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4b2cc1-f6af-4aec-8516-f08eb646c9a8" path="/var/lib/kubelet/pods/da4b2cc1-f6af-4aec-8516-f08eb646c9a8/volumes" Sep 09 16:10:01.190758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.190742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerDied","Data":"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324"} Sep 09 16:10:01.191055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.191042 2562 scope.go:117] "RemoveContainer" containerID="b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324" Sep 09 16:10:01.192011 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.191987 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:01.192074 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.192060 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:01.192116 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.192093 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:01.313008 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.312967 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:10:01.313560 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:01.313527 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" containerID="cri-o://c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080" gracePeriod=30 Sep 09 16:10:02.195828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.195790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerStarted","Data":"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d"} Sep 09 16:10:02.312789 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.312752 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:02.313749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.313721 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4b2cc1-f6af-4aec-8516-f08eb646c9a8" containerName="revoke-keys" Sep 09 16:10:02.313922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.313908 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4b2cc1-f6af-4aec-8516-f08eb646c9a8" containerName="revoke-keys" Sep 09 16:10:02.314193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.314175 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4b2cc1-f6af-4aec-8516-f08eb646c9a8" containerName="revoke-keys" Sep 09 16:10:02.316818 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.316797 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:02.316916 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.316905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.320055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.320026 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-dae90c4d-gw-openshift-default-dockercfg-2vlph\"" Sep 09 16:10:02.444790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpxn\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.444975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.444975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.444975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.444975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.445129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.444974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.445129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.445063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.445129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.445112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.445225 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.445192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546401 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546401 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546401 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpxn\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546962 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.546962 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.546842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.547055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.547036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.547177 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.547149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.547247 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.547156 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.547431 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.547387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.548994 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.548973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.549138 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.549123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.554294 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.554275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpxn\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.554294 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.554288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token\") pod \"e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.629672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.629636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:02.871914 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.871877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:10:02.872104 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.872080 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 09 16:10:02.872153 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.872127 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 09 16:10:02.924195 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.924169 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:02.926040 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:10:02.926012 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a8fe5a_fef8_4d9a_ac00_41606bc7dfb6.slice/crio-f0547c0483c992f3cee87daf0643d90cda14c8946910aaae0e7f69ed6f512429 WatchSource:0}: Error finding container f0547c0483c992f3cee87daf0643d90cda14c8946910aaae0e7f69ed6f512429: Status 404 returned error can't find the container with id f0547c0483c992f3cee87daf0643d90cda14c8946910aaae0e7f69ed6f512429 Sep 09 16:10:02.928496 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.928463 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:02.928589 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.928565 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:02.928662 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:02.928623 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:03.206479 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.206366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerStarted","Data":"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7"} Sep 09 16:10:03.206479 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.206449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerStarted","Data":"f0547c0483c992f3cee87daf0643d90cda14c8946910aaae0e7f69ed6f512429"} Sep 09 16:10:03.629786 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.629750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:03.634866 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.634836 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:03.653809 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.653769 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podStartSLOduration=1.6537551069999998 podStartE2EDuration="1.653755107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:03.258862494 +0000 UTC m=+1122.600054322" watchObservedRunningTime="2026-09-09 16:10:03.653755107 +0000 UTC m=+1122.994946936" Sep 09 16:10:03.871379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.871339 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 09 16:10:03.871585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:03.871428 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 09 16:10:04.131384 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:04.131351 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:10:04.213666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:04.213637 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:04.214480 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:04.214463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:04.871632 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:04.871597 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 09 16:10:04.871793 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:04.871653 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 09 16:10:05.871635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:05.871598 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 09 16:10:05.872005 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:05.871654 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 09 16:10:06.609875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.609738 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:10:06.699807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.699770 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.700304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.700284 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.700809 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.700790 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.700987 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.700971 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.701106 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.701091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.701287 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.701273 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.701402 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.701389 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.702514 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.702496 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.702692 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.702675 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fxrf\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf\") pod \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\" (UID: \"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218\") " Sep 09 16:10:06.703362 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.700226 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:06.703504 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.700711 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:06.703587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.701618 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:06.703669 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.702423 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:10:06.706941 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.706809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:10:06.706941 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.706911 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data" (OuterVolumeSpecName: "istio-data") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:06.707612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.707574 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:06.708391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.708359 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token" (OuterVolumeSpecName: "istio-token") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:10:06.708686 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.708663 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf" (OuterVolumeSpecName: "kube-api-access-8fxrf") pod "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" (UID: "014a766d-98a5-4f8a-8d1d-5d8cf2b2c218"). InnerVolumeSpecName "kube-api-access-8fxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:10:06.804533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804494 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804528 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804544 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804556 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804567 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804577 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804587 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxrf\" (UniqueName: \"kubernetes.io/projected/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-kube-api-access-8fxrf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804607 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.804790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.804619 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:06.871985 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.871891 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 09 16:10:06.871985 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:06.871964 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 09 16:10:07.232392 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.232356 2562 generic.go:356] "Generic (PLEG): container finished" podID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerID="c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080" exitCode=0 Sep 09 16:10:07.232585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.232426 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" Sep 09 16:10:07.232585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.232443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerDied","Data":"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080"} Sep 09 16:10:07.232585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.232516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k" event={"ID":"014a766d-98a5-4f8a-8d1d-5d8cf2b2c218","Type":"ContainerDied","Data":"09fb09986d68d241084986b1b42dcff7201826a60a6f292779cda7fda4e3e1f5"} Sep 09 16:10:07.232585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.232548 2562 scope.go:117] "RemoveContainer" containerID="c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080" Sep 09 16:10:07.242576 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.242552 2562 scope.go:117] "RemoveContainer" containerID="23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7" Sep 09 16:10:07.252589 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.252561 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:10:07.253559 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.253535 2562 scope.go:117] "RemoveContainer" containerID="c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080" Sep 09 16:10:07.253874 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:10:07.253852 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080\": container with ID starting with c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080 not found: ID does not exist" containerID="c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080" Sep 09 16:10:07.253923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.253889 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080"} err="failed to get container status \"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080\": rpc error: code = NotFound desc = could not find container \"c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080\": container with ID starting with c70ef2c3059d2552253abb0011328380c80e795190773d4ad54ed1422c8a3080 not found: ID does not exist" Sep 09 16:10:07.253923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.253915 2562 scope.go:117] "RemoveContainer" containerID="23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7" Sep 09 16:10:07.254210 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:10:07.254190 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7\": container with ID starting with 23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7 not found: ID does not exist" containerID="23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7" Sep 09 16:10:07.254279 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.254214 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7"} err="failed to get container status \"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7\": rpc error: code = NotFound desc = could not find container \"23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7\": container with ID starting with 23ebe799fff3dd4b2ebbc58d97c9b61c84bf880b36b430bca7d5c45f5b45fee7 not found: ID does not exist" Sep 09 16:10:07.255109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.255090 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-083463c6-openshift-default-68f9b5f6bb-5mw7k"] Sep 09 16:10:07.875507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.875473 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:10:07.875903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.875763 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:10:07.876502 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:07.876482 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:10:09.190261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:09.190224 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" path="/var/lib/kubelet/pods/014a766d-98a5-4f8a-8d1d-5d8cf2b2c218/volumes" Sep 09 16:10:17.536754 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:17.536715 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:10:22.327084 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.327050 2562 generic.go:356] "Generic (PLEG): container finished" podID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerID="894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7" exitCode=0 Sep 09 16:10:22.327517 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.327126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerDied","Data":"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7"} Sep 09 16:10:22.327584 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.327569 2562 scope.go:117] "RemoveContainer" containerID="894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7" Sep 09 16:10:22.328860 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.328826 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:22.328942 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.328914 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:22.328991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:22.328957 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:23.335323 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:23.335288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerStarted","Data":"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816"} Sep 09 16:10:23.630053 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:23.630018 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:23.630226 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:23.630208 2562 patch_prober.go:28] interesting pod/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 09 16:10:23.630272 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:23.630257 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 09 16:10:24.630490 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:24.630457 2562 patch_prober.go:28] interesting pod/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 09 16:10:24.630869 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:24.630511 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 09 16:10:25.630925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:25.630885 2562 patch_prober.go:28] interesting pod/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 09 16:10:25.631310 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:25.630942 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 09 16:10:26.630908 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:26.630871 2562 patch_prober.go:28] interesting pod/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 09 16:10:26.631078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:26.630929 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 09 16:10:27.630814 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:27.630766 2562 patch_prober.go:28] interesting pod/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 09 16:10:27.631031 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:27.630830 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 09 16:10:28.633485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:28.633441 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:28.633948 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:28.633743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:28.634516 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:28.634500 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:29.718815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.718780 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm"] Sep 09 16:10:29.719275 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719261 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.719327 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719277 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.719327 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719296 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.719327 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719304 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.719444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719421 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.719444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.719432 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="014a766d-98a5-4f8a-8d1d-5d8cf2b2c218" containerName="istio-proxy" Sep 09 16:10:29.721533 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.721517 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:29.724079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.724056 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:10:29.724207 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.724062 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:10:29.731763 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.731740 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm"] Sep 09 16:10:29.741843 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.741816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm\" (UID: \"d9e7fe5b-8470-403e-8c88-ed031082ea16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:29.843404 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.843361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm\" (UID: \"d9e7fe5b-8470-403e-8c88-ed031082ea16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:29.844009 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:29.843989 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm\" (UID: \"d9e7fe5b-8470-403e-8c88-ed031082ea16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:30.033071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:30.032988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:30.302252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:30.302226 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm"] Sep 09 16:10:30.304482 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:10:30.304454 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e7fe5b_8470_403e_8c88_ed031082ea16.slice/crio-ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440 WatchSource:0}: Error finding container ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440: Status 404 returned error can't find the container with id ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440 Sep 09 16:10:30.381469 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:30.381431 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" event={"ID":"d9e7fe5b-8470-403e-8c88-ed031082ea16","Type":"ContainerStarted","Data":"ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440"} Sep 09 16:10:31.389307 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:31.389269 2562 generic.go:356] "Generic (PLEG): container finished" podID="d9e7fe5b-8470-403e-8c88-ed031082ea16" containerID="be608f0fcffcee7911169d9319078554a0fe848ec10e9c1c7f2779e65bc36085" exitCode=0 Sep 09 16:10:31.389752 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:31.389325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" event={"ID":"d9e7fe5b-8470-403e-8c88-ed031082ea16","Type":"ContainerDied","Data":"be608f0fcffcee7911169d9319078554a0fe848ec10e9c1c7f2779e65bc36085"} Sep 09 16:10:32.527170 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:32.527145 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:32.571021 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:32.570983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca\") pod \"d9e7fe5b-8470-403e-8c88-ed031082ea16\" (UID: \"d9e7fe5b-8470-403e-8c88-ed031082ea16\") " Sep 09 16:10:32.571325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:32.571301 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9e7fe5b-8470-403e-8c88-ed031082ea16" (UID: "d9e7fe5b-8470-403e-8c88-ed031082ea16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:10:32.672158 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:32.672075 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e7fe5b-8470-403e-8c88-ed031082ea16-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:33.407435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:33.407379 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" Sep 09 16:10:33.407620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:33.407378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm" event={"ID":"d9e7fe5b-8470-403e-8c88-ed031082ea16","Type":"ContainerDied","Data":"ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440"} Sep 09 16:10:33.407620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:33.407565 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecae00850d9d1c0eaba1b9c826069944c64092776c85d88d2b9378819b2a7440" Sep 09 16:10:34.139070 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:34.139031 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:10:36.450099 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:36.450060 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm"] Sep 09 16:10:36.452374 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:36.452344 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dae90c4d-mw5mm"] Sep 09 16:10:36.588493 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:36.588457 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:36.588768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:36.588720 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" containerID="cri-o://e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816" gracePeriod=30 Sep 09 16:10:37.190527 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.190496 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e7fe5b-8470-403e-8c88-ed031082ea16" path="/var/lib/kubelet/pods/d9e7fe5b-8470-403e-8c88-ed031082ea16/volumes" Sep 09 16:10:37.375033 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.374989 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:10:37.375737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.375712 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e7fe5b-8470-403e-8c88-ed031082ea16" containerName="revoke-keys" Sep 09 16:10:37.375737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.375735 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e7fe5b-8470-403e-8c88-ed031082ea16" containerName="revoke-keys" Sep 09 16:10:37.375944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.375847 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e7fe5b-8470-403e-8c88-ed031082ea16" containerName="revoke-keys" Sep 09 16:10:37.378388 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.378366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.381330 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.381301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-58050ce1-openshift-default-dockercfg-hl5l6\"" Sep 09 16:10:37.395336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.395310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:10:37.421301 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421480 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421480 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421480 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxdf\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421480 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.421682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.421672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.522537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.522537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.522537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.522537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxdf\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.522916 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.523131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.523154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.523212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.523364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.523293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.524932 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.524907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.525032 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.525002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.532397 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.532371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.532943 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.532922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxdf\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf\") pod \"e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.691661 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.691606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:37.968603 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.968565 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:10:37.970352 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:10:37.970324 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1a05a1_b69a_4330_a039_3337c84e02c8.slice/crio-5e7dfd00513a858bb09d1c742307612c0731569c68b8009be2c49c9f5d2764f6 WatchSource:0}: Error finding container 5e7dfd00513a858bb09d1c742307612c0731569c68b8009be2c49c9f5d2764f6: Status 404 returned error can't find the container with id 5e7dfd00513a858bb09d1c742307612c0731569c68b8009be2c49c9f5d2764f6 Sep 09 16:10:37.972570 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.972536 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:37.972647 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.972606 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:37.972647 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:37.972635 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:38.441955 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:38.441913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerStarted","Data":"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3"} Sep 09 16:10:38.442140 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:38.441978 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerStarted","Data":"5e7dfd00513a858bb09d1c742307612c0731569c68b8009be2c49c9f5d2764f6"} Sep 09 16:10:38.467844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:38.467789 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podStartSLOduration=1.467773404 podStartE2EDuration="1.467773404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:10:38.465623751 +0000 UTC m=+1157.806815590" watchObservedRunningTime="2026-09-09 16:10:38.467773404 +0000 UTC m=+1157.808965233" Sep 09 16:10:38.691887 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:38.691854 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:39.697166 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:39.697132 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:40.453518 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:40.453479 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:40.454427 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:40.454389 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:10:41.864474 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.864390 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:41.979347 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979561 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979393 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979561 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkpxn\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979561 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979501 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979561 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979533 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979584 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979636 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979674 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.979832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979678 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:41.979832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979745 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token\") pod \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\" (UID: \"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6\") " Sep 09 16:10:41.980089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979960 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data" (OuterVolumeSpecName: "istio-data") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:41.980089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.979976 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:41.980191 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.980158 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:41.980191 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.980179 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:41.980297 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.980191 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:41.980356 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.980304 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:41.980486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.980460 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:10:41.982390 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.982354 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn" (OuterVolumeSpecName: "kube-api-access-qkpxn") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "kube-api-access-qkpxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:10:41.982691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.982666 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token" (OuterVolumeSpecName: "istio-token") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:10:41.982826 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.982803 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:10:41.982941 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:41.982923 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" (UID: "71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:10:42.081166 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081121 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.081166 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081165 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.081435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081183 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.081435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081198 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpxn\" (UniqueName: \"kubernetes.io/projected/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-kube-api-access-qkpxn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.081435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081213 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.081435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.081227 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:10:42.465883 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.465846 2562 generic.go:356] "Generic (PLEG): container finished" podID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerID="e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816" exitCode=0 Sep 09 16:10:42.466025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.465898 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" Sep 09 16:10:42.466025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.465931 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerDied","Data":"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816"} Sep 09 16:10:42.466025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.465987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg" event={"ID":"71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6","Type":"ContainerDied","Data":"f0547c0483c992f3cee87daf0643d90cda14c8946910aaae0e7f69ed6f512429"} Sep 09 16:10:42.466025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.466010 2562 scope.go:117] "RemoveContainer" containerID="e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816" Sep 09 16:10:42.475787 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.475766 2562 scope.go:117] "RemoveContainer" containerID="894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7" Sep 09 16:10:42.486587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.486567 2562 scope.go:117] "RemoveContainer" containerID="e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816" Sep 09 16:10:42.486897 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:10:42.486876 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816\": container with ID starting with e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816 not found: ID does not exist" containerID="e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816" Sep 09 16:10:42.486981 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.486908 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816"} err="failed to get container status \"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816\": rpc error: code = NotFound desc = could not find container \"e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816\": container with ID starting with e4a3c2e30a633bec0105f1496c1c0f55132f3fc68b11515d3ddd06a1b5995816 not found: ID does not exist" Sep 09 16:10:42.486981 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.486929 2562 scope.go:117] "RemoveContainer" containerID="894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7" Sep 09 16:10:42.487219 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:10:42.487197 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7\": container with ID starting with 894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7 not found: ID does not exist" containerID="894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7" Sep 09 16:10:42.487289 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.487227 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7"} err="failed to get container status \"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7\": rpc error: code = NotFound desc = could not find container \"894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7\": container with ID starting with 894167b00a1c9500f96cf926fc8073f666e1d900d460353c1c82e5fa4bccb5f7 not found: ID does not exist" Sep 09 16:10:42.494979 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.494956 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:42.504638 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:42.504615 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-dae90c4d-gw-openshift-default-784cb9bf4d-h9npg"] Sep 09 16:10:43.189942 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:43.189904 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" path="/var/lib/kubelet/pods/71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6/volumes" Sep 09 16:10:48.836571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:48.836530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:10:59.575132 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.575098 2562 generic.go:356] "Generic (PLEG): container finished" podID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerID="776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3" exitCode=0 Sep 09 16:10:59.575554 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.575137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerDied","Data":"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3"} Sep 09 16:10:59.575607 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.575551 2562 scope.go:117] "RemoveContainer" containerID="776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3" Sep 09 16:10:59.576611 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.576575 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:59.576672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.576653 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:10:59.576710 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:10:59.576685 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:00.584018 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:00.583974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerStarted","Data":"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e"} Sep 09 16:11:00.692632 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:00.692177 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:00.692632 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:00.692293 2562 patch_prober.go:28] interesting pod/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 09 16:11:00.692632 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:00.692348 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 09 16:11:01.692311 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:01.692266 2562 patch_prober.go:28] interesting pod/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 09 16:11:01.692836 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:01.692344 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 09 16:11:02.692172 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:02.692127 2562 patch_prober.go:28] interesting pod/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 09 16:11:02.692378 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:02.692192 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 09 16:11:03.692489 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.692446 2562 patch_prober.go:28] interesting pod/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 09 16:11:03.692986 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.692514 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 09 16:11:03.854155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.854109 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:11:03.855608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855577 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.855608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855603 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.855805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855626 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.855805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855632 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.855805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855709 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.855805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.855717 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a8fe5a-fef8-4d9a-ac00-41606bc7dfb6" containerName="istio-proxy" Sep 09 16:11:03.859075 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.859055 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.863159 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.863140 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a09c2a\"/\"autores-model-a09c2a-kserve-self-signed-certs\"" Sep 09 16:11:03.863271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.863140 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a09c2a\"/\"default-dockercfg-x9s84\"" Sep 09 16:11:03.863336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.863256 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a09c2a\"/\"kube-root-ca.crt\"" Sep 09 16:11:03.863336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.863140 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a09c2a\"/\"openshift-service-ca.crt\"" Sep 09 16:11:03.870829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.870808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:11:03.924935 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.924905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nnc\" (UniqueName: \"kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.925110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.924947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.925110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.924981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.925110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.925030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.925110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.925066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:03.925271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:03.925115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.025919 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.025832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.025919 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.025876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026124 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.025998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026124 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026093 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nnc\" (UniqueName: \"kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026185 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.026575 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.026515 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.028155 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.028128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.028562 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.028541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.034703 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.034678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nnc\" (UniqueName: \"kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc\") pod \"autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.170800 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.170762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:04.466063 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.466032 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:11:04.468134 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:11:04.468104 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab7b9dd2_31e4_4c9d_be75_0e1c1409ba7d.slice/crio-96d64ab23679b0bf7373cf7b6185fe37564ea9b0420f30bc0958ec47a7612633 WatchSource:0}: Error finding container 96d64ab23679b0bf7373cf7b6185fe37564ea9b0420f30bc0958ec47a7612633: Status 404 returned error can't find the container with id 96d64ab23679b0bf7373cf7b6185fe37564ea9b0420f30bc0958ec47a7612633 Sep 09 16:11:04.534951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.534924 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:11:04.623244 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.623210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" event={"ID":"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d","Type":"ContainerStarted","Data":"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d"} Sep 09 16:11:04.623458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.623251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" event={"ID":"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d","Type":"ContainerStarted","Data":"96d64ab23679b0bf7373cf7b6185fe37564ea9b0420f30bc0958ec47a7612633"} Sep 09 16:11:04.639745 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.639702 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" podStartSLOduration=1.639688898 podStartE2EDuration="1.639688898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:11:04.639026488 +0000 UTC m=+1183.980218318" watchObservedRunningTime="2026-09-09 16:11:04.639688898 +0000 UTC m=+1183.980880727" Sep 09 16:11:04.692690 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.692259 2562 patch_prober.go:28] interesting pod/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 09 16:11:04.692690 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:04.692314 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 09 16:11:05.614973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.614938 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r"] Sep 09 16:11:05.618749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.618728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:05.622056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.622038 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:11:05.622726 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.622697 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:11:05.633000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.632980 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r"] Sep 09 16:11:05.640819 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.640796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r\" (UID: \"7e608fb4-3ce2-48f6-9838-26ba931e9679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:05.741973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.741926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r\" (UID: \"7e608fb4-3ce2-48f6-9838-26ba931e9679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:05.742652 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.742626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r\" (UID: \"7e608fb4-3ce2-48f6-9838-26ba931e9679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:05.802959 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.802923 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:05.803318 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.803292 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:05.805054 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.805034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:05.931922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:05.931845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:06.203671 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:06.203648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r"] Sep 09 16:11:06.204966 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:11:06.204936 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e608fb4_3ce2_48f6_9838_26ba931e9679.slice/crio-37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc WatchSource:0}: Error finding container 37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc: Status 404 returned error can't find the container with id 37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc Sep 09 16:11:06.637504 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:06.637465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" event={"ID":"7e608fb4-3ce2-48f6-9838-26ba931e9679","Type":"ContainerStarted","Data":"536d5caa0a9f1e911e5370f908a7e421c9b25df614dbd7d4586f3e1102e1b7af"} Sep 09 16:11:06.637678 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:06.637517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" event={"ID":"7e608fb4-3ce2-48f6-9838-26ba931e9679","Type":"ContainerStarted","Data":"37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc"} Sep 09 16:11:06.654180 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:06.654132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" podStartSLOduration=1.654115336 podStartE2EDuration="1.654115336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:11:06.652355334 +0000 UTC m=+1185.993547164" watchObservedRunningTime="2026-09-09 16:11:06.654115336 +0000 UTC m=+1185.995307204" Sep 09 16:11:07.645671 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:07.645633 2562 generic.go:356] "Generic (PLEG): container finished" podID="7e608fb4-3ce2-48f6-9838-26ba931e9679" containerID="536d5caa0a9f1e911e5370f908a7e421c9b25df614dbd7d4586f3e1102e1b7af" exitCode=0 Sep 09 16:11:07.646072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:07.645722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" event={"ID":"7e608fb4-3ce2-48f6-9838-26ba931e9679","Type":"ContainerDied","Data":"536d5caa0a9f1e911e5370f908a7e421c9b25df614dbd7d4586f3e1102e1b7af"} Sep 09 16:11:08.794224 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:08.794196 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:08.873864 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:08.873831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca\") pod \"7e608fb4-3ce2-48f6-9838-26ba931e9679\" (UID: \"7e608fb4-3ce2-48f6-9838-26ba931e9679\") " Sep 09 16:11:08.874168 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:08.874143 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e608fb4-3ce2-48f6-9838-26ba931e9679" (UID: "7e608fb4-3ce2-48f6-9838-26ba931e9679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:11:08.975300 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:08.975231 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e608fb4-3ce2-48f6-9838-26ba931e9679-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:09.658282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:09.658249 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" Sep 09 16:11:09.658472 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:09.658253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r" event={"ID":"7e608fb4-3ce2-48f6-9838-26ba931e9679","Type":"ContainerDied","Data":"37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc"} Sep 09 16:11:09.658472 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:09.658378 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37103d98f37a1564bab3b20d86906f2fdc996c295727168733d11c0955c9a4bc" Sep 09 16:11:12.028655 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:12.028619 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r"] Sep 09 16:11:12.032382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:12.032353 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58050ce1-c7v6r"] Sep 09 16:11:13.192720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:13.192682 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e608fb4-3ce2-48f6-9838-26ba931e9679" path="/var/lib/kubelet/pods/7e608fb4-3ce2-48f6-9838-26ba931e9679/volumes" Sep 09 16:11:13.213013 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:13.212979 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:11:13.213179 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:13.213159 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" containerID="cri-o://8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e" gracePeriod=30 Sep 09 16:11:14.080345 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.080313 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:14.081091 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.081070 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e608fb4-3ce2-48f6-9838-26ba931e9679" containerName="revoke-keys" Sep 09 16:11:14.081143 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.081097 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e608fb4-3ce2-48f6-9838-26ba931e9679" containerName="revoke-keys" Sep 09 16:11:14.081207 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.081195 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e608fb4-3ce2-48f6-9838-26ba931e9679" containerName="revoke-keys" Sep 09 16:11:14.084933 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.084915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.088972 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.088949 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f772da8e-openshift-default-dockercfg-qnzj5\"" Sep 09 16:11:14.104997 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.104973 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:14.171337 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.171306 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:14.171599 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.171584 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:14.190920 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.190895 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:14.230453 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvr9s\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.230944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.230908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331601 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvr9s\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331601 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.331806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332023 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332023 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332023 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.331930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332176 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.332064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332176 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.332064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332254 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.332240 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332293 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.332276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.332495 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.332470 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.334051 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.334028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.334252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.334235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.343324 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.343303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvr9s\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.344431 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.344393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token\") pod \"e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.395147 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.395112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:14.675958 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.675933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:14.677269 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:11:14.677238 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04274c0_03a9_4a76_8b37_304b59c8b295.slice/crio-ab231911f5147a0f8e292a25aec5c1e9e175687605af5fc7800311a3a8339eb4 WatchSource:0}: Error finding container ab231911f5147a0f8e292a25aec5c1e9e175687605af5fc7800311a3a8339eb4: Status 404 returned error can't find the container with id ab231911f5147a0f8e292a25aec5c1e9e175687605af5fc7800311a3a8339eb4 Sep 09 16:11:14.679623 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.679583 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:14.679699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.679671 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:14.679744 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.679713 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:14.692386 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.692360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerStarted","Data":"ab231911f5147a0f8e292a25aec5c1e9e175687605af5fc7800311a3a8339eb4"} Sep 09 16:11:14.704649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:14.704628 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:15.700695 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:15.700655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerStarted","Data":"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd"} Sep 09 16:11:15.728720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:15.728672 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podStartSLOduration=1.728655781 podStartE2EDuration="1.728655781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:11:15.726899878 +0000 UTC m=+1195.068091778" watchObservedRunningTime="2026-09-09 16:11:15.728655781 +0000 UTC m=+1195.069847609" Sep 09 16:11:16.396185 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:16.396144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:16.401072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:16.401041 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:16.706128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:16.706045 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:16.707071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:16.707053 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:18.476837 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.476775 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:18.587624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587592 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587769 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587685 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587769 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587706 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587769 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587727 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587885 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587945 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.587997 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.587986 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.588057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588023 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.588057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588023 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:18.588057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588052 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srxdf\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf\") pod \"6c1a05a1-b69a-4330-a039-3337c84e02c8\" (UID: \"6c1a05a1-b69a-4330-a039-3337c84e02c8\") " Sep 09 16:11:18.588217 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588115 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:18.588217 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588133 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:11:18.588340 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588226 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:18.588395 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588350 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:18.588626 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588602 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a05a1-b69a-4330-a039-3337c84e02c8-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.588691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588630 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.588691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588646 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.588691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588660 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.588691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.588672 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.589912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.589888 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:18.590029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.590008 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:11:18.590122 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.590104 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:11:18.590172 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.590121 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf" (OuterVolumeSpecName: "kube-api-access-srxdf") pod "6c1a05a1-b69a-4330-a039-3337c84e02c8" (UID: "6c1a05a1-b69a-4330-a039-3337c84e02c8"). InnerVolumeSpecName "kube-api-access-srxdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:11:18.689646 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.689605 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.689646 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.689637 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.689646 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.689650 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a05a1-b69a-4330-a039-3337c84e02c8-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.689869 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.689661 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxdf\" (UniqueName: \"kubernetes.io/projected/6c1a05a1-b69a-4330-a039-3337c84e02c8-kube-api-access-srxdf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:18.718954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.718917 2562 generic.go:356] "Generic (PLEG): container finished" podID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerID="8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e" exitCode=0 Sep 09 16:11:18.719148 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.718966 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" Sep 09 16:11:18.719148 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.719010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerDied","Data":"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e"} Sep 09 16:11:18.719148 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.719083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj" event={"ID":"6c1a05a1-b69a-4330-a039-3337c84e02c8","Type":"ContainerDied","Data":"5e7dfd00513a858bb09d1c742307612c0731569c68b8009be2c49c9f5d2764f6"} Sep 09 16:11:18.719148 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.719113 2562 scope.go:117] "RemoveContainer" containerID="8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e" Sep 09 16:11:18.729060 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.729042 2562 scope.go:117] "RemoveContainer" containerID="776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3" Sep 09 16:11:18.737744 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.737726 2562 scope.go:117] "RemoveContainer" containerID="8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e" Sep 09 16:11:18.738050 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:18.738027 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e\": container with ID starting with 8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e not found: ID does not exist" containerID="8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e" Sep 09 16:11:18.738132 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.738058 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e"} err="failed to get container status \"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e\": rpc error: code = NotFound desc = could not find container \"8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e\": container with ID starting with 8d9932fe0b824fb41d96fc0f14205f49c3f0696ad4e690c53147f78f05e28f3e not found: ID does not exist" Sep 09 16:11:18.738132 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.738076 2562 scope.go:117] "RemoveContainer" containerID="776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3" Sep 09 16:11:18.738390 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:18.738371 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3\": container with ID starting with 776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3 not found: ID does not exist" containerID="776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3" Sep 09 16:11:18.738540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.738393 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3"} err="failed to get container status \"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3\": rpc error: code = NotFound desc = could not find container \"776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3\": container with ID starting with 776e6a1cd4e3ec6372b6fe0d096b2114a2d0d1e1e342246fc50aed828142abb3 not found: ID does not exist" Sep 09 16:11:18.748082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.748052 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:11:18.762381 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.762346 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-58050ce1-openshift-default-55cb49df46-4hzbj"] Sep 09 16:11:18.963828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:18.963788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:11:19.191829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:19.191799 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" path="/var/lib/kubelet/pods/6c1a05a1-b69a-4330-a039-3337c84e02c8/volumes" Sep 09 16:11:23.681691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:23.681659 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:11:23.684259 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:23.684234 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:11:34.736900 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:34.736865 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:11:38.130282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.130246 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2"] Sep 09 16:11:38.130828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.130713 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:38.130828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.130725 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:38.130828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.130802 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:38.130828 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.130810 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:38.134016 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.133997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:38.137118 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.136775 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:11:38.137918 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.137892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:11:38.141183 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.141130 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2"] Sep 09 16:11:38.193453 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.193419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2\" (UID: \"0b5195d4-7869-439c-8f22-efe6da0d73bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:38.294644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.294613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2\" (UID: \"0b5195d4-7869-439c-8f22-efe6da0d73bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:38.295221 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.295202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2\" (UID: \"0b5195d4-7869-439c-8f22-efe6da0d73bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:38.447761 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.447731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:38.722514 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.722487 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2"] Sep 09 16:11:38.723922 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:11:38.723901 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5195d4_7869_439c_8f22_efe6da0d73bd.slice/crio-832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70 WatchSource:0}: Error finding container 832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70: Status 404 returned error can't find the container with id 832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70 Sep 09 16:11:38.846138 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.846107 2562 generic.go:356] "Generic (PLEG): container finished" podID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerID="0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd" exitCode=0 Sep 09 16:11:38.846291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.846179 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerDied","Data":"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd"} Sep 09 16:11:38.846719 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.846698 2562 scope.go:117] "RemoveContainer" containerID="0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd" Sep 09 16:11:38.847634 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.847613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" event={"ID":"0b5195d4-7869-439c-8f22-efe6da0d73bd","Type":"ContainerStarted","Data":"832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70"} Sep 09 16:11:38.847922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.847891 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:38.847999 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.847983 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:38.848036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:38.848021 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:11:39.855956 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:39.855916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerStarted","Data":"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30"} Sep 09 16:11:39.857402 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:39.857369 2562 generic.go:356] "Generic (PLEG): container finished" podID="0b5195d4-7869-439c-8f22-efe6da0d73bd" containerID="cc863a47990022fbd89069f0314851191d64ed5c74b0051c1b9d9791602a9467" exitCode=0 Sep 09 16:11:39.857551 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:39.857455 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" event={"ID":"0b5195d4-7869-439c-8f22-efe6da0d73bd","Type":"ContainerDied","Data":"cc863a47990022fbd89069f0314851191d64ed5c74b0051c1b9d9791602a9467"} Sep 09 16:11:40.395933 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:40.395892 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:40.396147 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:40.396027 2562 patch_prober.go:28] interesting pod/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 09 16:11:40.396147 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:40.396082 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 09 16:11:41.037553 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.037526 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:41.125182 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.125094 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca\") pod \"0b5195d4-7869-439c-8f22-efe6da0d73bd\" (UID: \"0b5195d4-7869-439c-8f22-efe6da0d73bd\") " Sep 09 16:11:41.125486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.125464 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b5195d4-7869-439c-8f22-efe6da0d73bd" (UID: "0b5195d4-7869-439c-8f22-efe6da0d73bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:11:41.226531 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.226502 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5195d4-7869-439c-8f22-efe6da0d73bd-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:41.395973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.395867 2562 patch_prober.go:28] interesting pod/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 09 16:11:41.395973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.395948 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 09 16:11:41.869820 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.869778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" event={"ID":"0b5195d4-7869-439c-8f22-efe6da0d73bd","Type":"ContainerDied","Data":"832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70"} Sep 09 16:11:41.869975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.869842 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832b07e2b10cea596ef81c7a04b7e37e181428b8b8db521b01a07fa593325b70" Sep 09 16:11:41.869975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.869796 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2" Sep 09 16:11:41.993745 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.993701 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg"] Sep 09 16:11:41.994849 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.994825 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5195d4-7869-439c-8f22-efe6da0d73bd" containerName="revoke-keys" Sep 09 16:11:41.995038 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.995024 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5195d4-7869-439c-8f22-efe6da0d73bd" containerName="revoke-keys" Sep 09 16:11:41.995233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.995220 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:41.995347 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.995335 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a05a1-b69a-4330-a039-3337c84e02c8" containerName="istio-proxy" Sep 09 16:11:41.995605 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.995579 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5195d4-7869-439c-8f22-efe6da0d73bd" containerName="revoke-keys" Sep 09 16:11:41.998427 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:41.998389 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:42.001776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.001754 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:11:42.002144 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.002127 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:11:42.011458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.011421 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg"] Sep 09 16:11:42.033522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.033489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f772da8e-npncg\" (UID: \"8456de8b-92f6-4dbc-a3c7-11fe2582f370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:42.134453 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.134344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f772da8e-npncg\" (UID: \"8456de8b-92f6-4dbc-a3c7-11fe2582f370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:42.135015 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.134992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f772da8e-npncg\" (UID: \"8456de8b-92f6-4dbc-a3c7-11fe2582f370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:42.174672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.174639 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:11:42.313600 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.313563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:42.402967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.396233 2562 patch_prober.go:28] interesting pod/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 09 16:11:42.402967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.396294 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 09 16:11:42.600930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.600883 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg"] Sep 09 16:11:42.612059 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:11:42.612026 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8456de8b_92f6_4dbc_a3c7_11fe2582f370.slice/crio-191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d WatchSource:0}: Error finding container 191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d: Status 404 returned error can't find the container with id 191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d Sep 09 16:11:42.876889 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:42.876856 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" event={"ID":"8456de8b-92f6-4dbc-a3c7-11fe2582f370","Type":"ContainerStarted","Data":"191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d"} Sep 09 16:11:43.395737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:43.395673 2562 patch_prober.go:28] interesting pod/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 09 16:11:43.396238 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:43.395736 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 09 16:11:43.887179 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:43.887136 2562 generic.go:356] "Generic (PLEG): container finished" podID="8456de8b-92f6-4dbc-a3c7-11fe2582f370" containerID="ab676a826fc5955e3f5ea7dd41c742107d7ddde744fa039728f554e307f2ef8f" exitCode=0 Sep 09 16:11:43.887386 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:43.887224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" event={"ID":"8456de8b-92f6-4dbc-a3c7-11fe2582f370","Type":"ContainerDied","Data":"ab676a826fc5955e3f5ea7dd41c742107d7ddde744fa039728f554e307f2ef8f"} Sep 09 16:11:44.396428 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:44.396361 2562 patch_prober.go:28] interesting pod/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 09 16:11:44.396923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:44.396450 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 09 16:11:44.643619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:44.643589 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:11:45.049189 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.049168 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:45.166991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.166901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca\") pod \"8456de8b-92f6-4dbc-a3c7-11fe2582f370\" (UID: \"8456de8b-92f6-4dbc-a3c7-11fe2582f370\") " Sep 09 16:11:45.167349 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.167317 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8456de8b-92f6-4dbc-a3c7-11fe2582f370" (UID: "8456de8b-92f6-4dbc-a3c7-11fe2582f370"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:11:45.268540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.268502 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8456de8b-92f6-4dbc-a3c7-11fe2582f370-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:45.399525 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.399493 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:45.399991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.399901 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:45.400512 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.400492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:45.903245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.903218 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" Sep 09 16:11:45.903400 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.903252 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg" event={"ID":"8456de8b-92f6-4dbc-a3c7-11fe2582f370","Type":"ContainerDied","Data":"191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d"} Sep 09 16:11:45.903400 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:45.903295 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191814a8a0989de0f4db6e8583cf7f5743739d86c30fbfbbf5ccdc4fa5cc6a4d" Sep 09 16:11:49.098496 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.098460 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg"] Sep 09 16:11:49.114758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.114725 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f772da8e-npncg"] Sep 09 16:11:49.192795 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.192765 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8456de8b-92f6-4dbc-a3c7-11fe2582f370" path="/var/lib/kubelet/pods/8456de8b-92f6-4dbc-a3c7-11fe2582f370/volumes" Sep 09 16:11:49.201734 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.201705 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:49.201953 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.201923 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" containerID="cri-o://fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30" gracePeriod=30 Sep 09 16:11:49.287527 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.287461 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2"] Sep 09 16:11:49.289659 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:49.289635 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a09c2a-h2pc2"] Sep 09 16:11:51.190367 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:51.190323 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5195d4-7869-439c-8f22-efe6da0d73bd" path="/var/lib/kubelet/pods/0b5195d4-7869-439c-8f22-efe6da0d73bd/volumes" Sep 09 16:11:54.226170 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.226142 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:11:54.487902 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.487875 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:54.572894 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.572863 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573046 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.572900 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvr9s\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573046 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.572950 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573046 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573011 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573076 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573111 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573151 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573209 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo\") pod \"c04274c0-03a9-4a76-8b37-304b59c8b295\" (UID: \"c04274c0-03a9-4a76-8b37-304b59c8b295\") " Sep 09 16:11:54.573546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573241 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:54.573620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573563 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:11:54.573620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573563 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:54.573620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573585 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data" (OuterVolumeSpecName: "istio-data") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:54.573779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573760 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.573815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573786 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04274c0-03a9-4a76-8b37-304b59c8b295-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.573815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573800 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.573894 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573815 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.573894 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.573828 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.575267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.575244 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s" (OuterVolumeSpecName: "kube-api-access-kvr9s") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "kube-api-access-kvr9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:11:54.575375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.575262 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:11:54.575375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.575269 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token" (OuterVolumeSpecName: "istio-token") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:11:54.575543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.575516 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c04274c0-03a9-4a76-8b37-304b59c8b295" (UID: "c04274c0-03a9-4a76-8b37-304b59c8b295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:54.674755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.674722 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.674755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.674748 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvr9s\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-kube-api-access-kvr9s\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.674755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.674758 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.674755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.674766 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04274c0-03a9-4a76-8b37-304b59c8b295-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:54.955935 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.955901 2562 generic.go:356] "Generic (PLEG): container finished" podID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerID="fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30" exitCode=0 Sep 09 16:11:54.956079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.955952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" Sep 09 16:11:54.956079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.955981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerDied","Data":"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30"} Sep 09 16:11:54.956079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.956040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk" event={"ID":"c04274c0-03a9-4a76-8b37-304b59c8b295","Type":"ContainerDied","Data":"ab231911f5147a0f8e292a25aec5c1e9e175687605af5fc7800311a3a8339eb4"} Sep 09 16:11:54.956079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.956072 2562 scope.go:117] "RemoveContainer" containerID="fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30" Sep 09 16:11:54.965742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.965721 2562 scope.go:117] "RemoveContainer" containerID="0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd" Sep 09 16:11:54.974685 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.974668 2562 scope.go:117] "RemoveContainer" containerID="fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30" Sep 09 16:11:54.974950 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:54.974927 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30\": container with ID starting with fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30 not found: ID does not exist" containerID="fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30" Sep 09 16:11:54.975003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.974958 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30"} err="failed to get container status \"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30\": rpc error: code = NotFound desc = could not find container \"fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30\": container with ID starting with fa8e6da178877b76432b05200ea2faad374890cf3a038f2197fc2c498b185c30 not found: ID does not exist" Sep 09 16:11:54.975003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.974975 2562 scope.go:117] "RemoveContainer" containerID="0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd" Sep 09 16:11:54.975223 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:54.975199 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd\": container with ID starting with 0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd not found: ID does not exist" containerID="0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd" Sep 09 16:11:54.975368 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.975228 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd"} err="failed to get container status \"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd\": rpc error: code = NotFound desc = could not find container \"0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd\": container with ID starting with 0655b35aecde0019203f2bfe18fb375fc3a35971fe53bd6f17110f0fb7cad6bd not found: ID does not exist" Sep 09 16:11:54.978440 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.978419 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:54.983606 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:54.983586 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f772da8e-openshift-default-544465cd44-9ntmk"] Sep 09 16:11:55.191599 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:55.191564 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" path="/var/lib/kubelet/pods/c04274c0-03a9-4a76-8b37-304b59c8b295/volumes" Sep 09 16:11:57.227959 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.227915 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" podUID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" containerName="main" containerID="cri-o://a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324" gracePeriod=45 Sep 09 16:11:57.486112 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.486043 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:11:57.604078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604046 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604142 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604241 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604283 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9mmm\" (UniqueName: \"kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604321 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604351 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir\") pod \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\" (UID: \"99005dc3-a71e-4c41-8bfd-b8a4ba37933c\") " Sep 09 16:11:57.604379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604357 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache" (OuterVolumeSpecName: "model-cache") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:57.604738 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604664 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home" (OuterVolumeSpecName: "home") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:57.604738 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604714 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:57.604893 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604827 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.604893 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604840 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.604893 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.604872 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.606305 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.606284 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:11:57.606718 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.606693 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm" (OuterVolumeSpecName: "dshm") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:11:57.606789 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.606697 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm" (OuterVolumeSpecName: "kube-api-access-k9mmm") pod "99005dc3-a71e-4c41-8bfd-b8a4ba37933c" (UID: "99005dc3-a71e-4c41-8bfd-b8a4ba37933c"). InnerVolumeSpecName "kube-api-access-k9mmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:11:57.705696 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.705670 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.705696 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.705695 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mmm\" (UniqueName: \"kubernetes.io/projected/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-kube-api-access-k9mmm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.705875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.705705 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99005dc3-a71e-4c41-8bfd-b8a4ba37933c-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:11:57.975094 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.975061 2562 generic.go:356] "Generic (PLEG): container finished" podID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" containerID="a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324" exitCode=0 Sep 09 16:11:57.975239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.975102 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" Sep 09 16:11:57.975239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.975113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" event={"ID":"99005dc3-a71e-4c41-8bfd-b8a4ba37933c","Type":"ContainerDied","Data":"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324"} Sep 09 16:11:57.975239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.975150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb" event={"ID":"99005dc3-a71e-4c41-8bfd-b8a4ba37933c","Type":"ContainerDied","Data":"8b4ef8f2c00353d09dcf1a3965f7a10f5fe92c88e6f9e4fe548eced5f6b00e1c"} Sep 09 16:11:57.975239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.975181 2562 scope.go:117] "RemoveContainer" containerID="a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324" Sep 09 16:11:57.984704 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.984683 2562 scope.go:117] "RemoveContainer" containerID="a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324" Sep 09 16:11:57.984964 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:57.984944 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324\": container with ID starting with a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324 not found: ID does not exist" containerID="a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324" Sep 09 16:11:57.985012 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.984972 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324"} err="failed to get container status \"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324\": rpc error: code = NotFound desc = could not find container \"a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324\": container with ID starting with a574fe4e5fc46513215ac50956e6b375af7b29f6b741ecf567b885353dec1324 not found: ID does not exist" Sep 09 16:11:57.998343 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:57.998321 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:11:58.002211 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:58.002192 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-049e4f89-kserve-694b46d496-7rtzb"] Sep 09 16:11:59.191481 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.191448 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" path="/var/lib/kubelet/pods/99005dc3-a71e-4c41-8bfd-b8a4ba37933c/volumes" Sep 09 16:11:59.663613 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.663549 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" podUID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" containerName="main" containerID="cri-o://23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d" gracePeriod=45 Sep 09 16:11:59.924813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.924756 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:59.989299 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.989270 2562 generic.go:356] "Generic (PLEG): container finished" podID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" containerID="23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d" exitCode=0 Sep 09 16:11:59.989469 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.989308 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" Sep 09 16:11:59.989469 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.989348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" event={"ID":"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d","Type":"ContainerDied","Data":"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d"} Sep 09 16:11:59.989469 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.989395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj" event={"ID":"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d","Type":"ContainerDied","Data":"96d64ab23679b0bf7373cf7b6185fe37564ea9b0420f30bc0958ec47a7612633"} Sep 09 16:11:59.989469 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.989439 2562 scope.go:117] "RemoveContainer" containerID="23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d" Sep 09 16:11:59.998224 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.998204 2562 scope.go:117] "RemoveContainer" containerID="23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d" Sep 09 16:11:59.998497 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:11:59.998478 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d\": container with ID starting with 23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d not found: ID does not exist" containerID="23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d" Sep 09 16:11:59.998547 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:11:59.998506 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d"} err="failed to get container status \"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d\": rpc error: code = NotFound desc = could not find container \"23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d\": container with ID starting with 23594e24baf5c7b972312eb7f445eacc3bee0e4aa730a83f7c8627b4ecea854d not found: ID does not exist" Sep 09 16:12:00.029758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029729 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.029903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029774 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.029903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029804 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.029903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029882 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2nnc\" (UniqueName: \"kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.030038 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029926 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home" (OuterVolumeSpecName: "home") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:00.030038 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.029941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.030038 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030015 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir\") pod \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\" (UID: \"ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d\") " Sep 09 16:12:00.030190 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:00.030355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030329 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:00.030495 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030477 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.030555 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030501 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.030555 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.030516 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.032208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.032176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:12:00.032424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.032391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm" (OuterVolumeSpecName: "dshm") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:00.032492 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.032434 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc" (OuterVolumeSpecName: "kube-api-access-b2nnc") pod "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" (UID: "ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d"). InnerVolumeSpecName "kube-api-access-b2nnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:12:00.131548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.131512 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2nnc\" (UniqueName: \"kubernetes.io/projected/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-kube-api-access-b2nnc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.131548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.131545 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.131548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.131555 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:00.323205 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.323175 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:12:00.334144 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:00.334116 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a09c2a/autores-model-a09c2a-kserve-7cfdd86d4c-f9vzj"] Sep 09 16:12:01.191118 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:01.191081 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" path="/var/lib/kubelet/pods/ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d/volumes" Sep 09 16:12:06.940188 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:06.940151 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:12:21.932885 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:21.932851 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:12:32.302038 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:32.302004 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:12:32.302459 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:32.302223 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" containerID="cri-o://46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621" gracePeriod=30 Sep 09 16:12:33.040118 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040088 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:12:33.040557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040542 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" containerName="main" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040558 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" containerName="main" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040568 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040574 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040581 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040587 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040600 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8456de8b-92f6-4dbc-a3c7-11fe2582f370" containerName="revoke-keys" Sep 09 16:12:33.040620 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040613 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8456de8b-92f6-4dbc-a3c7-11fe2582f370" containerName="revoke-keys" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040626 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" containerName="main" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040630 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" containerName="main" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040694 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040701 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7b9dd2-31e4-4c9d-be75-0e1c1409ba7d" containerName="main" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040708 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005dc3-a71e-4c41-8bfd-b8a4ba37933c" containerName="main" Sep 09 16:12:33.040912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.040716 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8456de8b-92f6-4dbc-a3c7-11fe2582f370" containerName="revoke-keys" Sep 09 16:12:33.045544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.045522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.048379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.048345 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-bfb318-openshift-default-dockercfg-wlsm8\"" Sep 09 16:12:33.056078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.056053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:12:33.145717 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.145922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.145922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkr2\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.145922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.146089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.146089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.145977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.146089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.146001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.146227 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.146078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.146227 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.146131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247668 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.247859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248146 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248146 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.247957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248146 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248146 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkr2\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248146 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248388 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.248839 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.248814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.250065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.250041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.250242 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.250221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.256122 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.256099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.256294 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.256274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkr2\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2\") pod \"e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.357477 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.357442 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:33.636642 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.636614 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:12:33.638665 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:12:33.638631 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1486de15_9f13_4fbd_bb7a_d212f72a891c.slice/crio-cb87a53383fcaef64ca88d0b353a96e2b2f37b4df8c604c1ed45c70fba99cd8c WatchSource:0}: Error finding container cb87a53383fcaef64ca88d0b353a96e2b2f37b4df8c604c1ed45c70fba99cd8c: Status 404 returned error can't find the container with id cb87a53383fcaef64ca88d0b353a96e2b2f37b4df8c604c1ed45c70fba99cd8c Sep 09 16:12:33.640932 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.640897 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:33.641022 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.640964 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:33.641022 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:33.640994 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:34.196484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:34.196453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerStarted","Data":"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713"} Sep 09 16:12:34.196661 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:34.196503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerStarted","Data":"cb87a53383fcaef64ca88d0b353a96e2b2f37b4df8c604c1ed45c70fba99cd8c"} Sep 09 16:12:34.216919 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:34.216872 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podStartSLOduration=1.216857942 podStartE2EDuration="1.216857942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:12:34.21427514 +0000 UTC m=+1273.555467006" watchObservedRunningTime="2026-09-09 16:12:34.216857942 +0000 UTC m=+1273.558049770" Sep 09 16:12:34.358451 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:34.358400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:34.363139 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:34.363113 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:35.205989 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:35.205955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:35.208912 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:35.208883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:37.565260 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.565210 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:12:37.693907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.693869 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.693907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.693911 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.693945 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.693966 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694000 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694087 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694147 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694220 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czcms\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694286 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket\") pod \"1935ec5f-226d-4eb7-9292-6343c8c69da7\" (UID: \"1935ec5f-226d-4eb7-9292-6343c8c69da7\") " Sep 09 16:12:37.694464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694340 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:37.694622 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694468 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:12:37.694622 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694540 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:37.694622 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694591 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data" (OuterVolumeSpecName: "istio-data") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:37.694781 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694704 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:37.694837 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694807 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.694837 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694824 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.694837 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694834 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.694998 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694841 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.694998 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.694852 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1935ec5f-226d-4eb7-9292-6343c8c69da7-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.696207 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.696182 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token" (OuterVolumeSpecName: "istio-token") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:12:37.696322 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.696207 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:12:37.696644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.696621 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:12:37.696644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.696631 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms" (OuterVolumeSpecName: "kube-api-access-czcms") pod "1935ec5f-226d-4eb7-9292-6343c8c69da7" (UID: "1935ec5f-226d-4eb7-9292-6343c8c69da7"). InnerVolumeSpecName "kube-api-access-czcms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:12:37.795716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.795676 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.795716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.795711 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.795716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.795723 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1935ec5f-226d-4eb7-9292-6343c8c69da7-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.795953 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.795734 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czcms\" (UniqueName: \"kubernetes.io/projected/1935ec5f-226d-4eb7-9292-6343c8c69da7-kube-api-access-czcms\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:12:37.837171 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:37.837089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:12:38.224404 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.224359 2562 generic.go:356] "Generic (PLEG): container finished" podID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerID="46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621" exitCode=0 Sep 09 16:12:38.224642 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.224438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" Sep 09 16:12:38.224642 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.224446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerDied","Data":"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621"} Sep 09 16:12:38.224642 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.224514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd" event={"ID":"1935ec5f-226d-4eb7-9292-6343c8c69da7","Type":"ContainerDied","Data":"24f5c9305b2108095ba740b0a4abdffb2806cdd632365b59dfdc08d79d761a0d"} Sep 09 16:12:38.224642 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.224544 2562 scope.go:117] "RemoveContainer" containerID="46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621" Sep 09 16:12:38.234667 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.234649 2562 scope.go:117] "RemoveContainer" containerID="cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db" Sep 09 16:12:38.242990 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.242974 2562 scope.go:117] "RemoveContainer" containerID="46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621" Sep 09 16:12:38.243253 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:12:38.243235 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621\": container with ID starting with 46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621 not found: ID does not exist" containerID="46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621" Sep 09 16:12:38.243325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.243264 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621"} err="failed to get container status \"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621\": rpc error: code = NotFound desc = could not find container \"46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621\": container with ID starting with 46e61ff2d57d443582540947a4907c1a21c72de15163e0e41fbe4916ae511621 not found: ID does not exist" Sep 09 16:12:38.243325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.243287 2562 scope.go:117] "RemoveContainer" containerID="cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db" Sep 09 16:12:38.243596 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:12:38.243578 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db\": container with ID starting with cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db not found: ID does not exist" containerID="cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db" Sep 09 16:12:38.243661 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.243603 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db"} err="failed to get container status \"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db\": rpc error: code = NotFound desc = could not find container \"cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db\": container with ID starting with cc4c9be4297855dc85eb0b9b046f3cdcc93719ccabe0e51408137ca4f46413db not found: ID does not exist" Sep 09 16:12:38.246483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.246463 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:12:38.252784 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:38.252757 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a09c2a-openshift-default-564444cf78-wvqhd"] Sep 09 16:12:39.192877 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:39.192833 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" path="/var/lib/kubelet/pods/1935ec5f-226d-4eb7-9292-6343c8c69da7/volumes" Sep 09 16:12:41.738815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:41.738780 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:12:49.927199 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:49.927162 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:12:51.247600 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:51.247555 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:12:51.248061 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:51.247631 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:12:53.319126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.319023 2562 generic.go:356] "Generic (PLEG): container finished" podID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerID="2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713" exitCode=0 Sep 09 16:12:53.319126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.319102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerDied","Data":"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713"} Sep 09 16:12:53.319584 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.319570 2562 scope.go:117] "RemoveContainer" containerID="2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713" Sep 09 16:12:53.320637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.320603 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:53.320709 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.320674 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:53.320709 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:53.320706 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:12:54.327036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:54.327002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerStarted","Data":"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5"} Sep 09 16:12:54.357936 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:54.357898 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:12:54.358128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:54.358083 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:54.358216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:54.358128 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:12:55.358574 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:55.358534 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:55.358958 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:55.358590 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:12:56.358132 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:56.358095 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:56.358306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:56.358150 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:12:57.358742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:57.358700 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:57.359215 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:57.358761 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:12:58.358451 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:58.358395 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:58.358644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:58.358472 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:12:59.357898 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:59.357865 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 16:12:59.358276 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:12:59.357915 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 16:13:00.362047 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:00.362016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:13:00.362402 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:00.362329 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:13:00.362933 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:00.362912 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:13:04.328909 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:04.328860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:13:08.847741 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.847701 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:08.848404 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848383 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848423 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848447 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848456 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848599 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848586 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848606 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1935ec5f-226d-4eb7-9292-6343c8c69da7" containerName="istio-proxy" Sep 09 16:13:08.848656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.848622 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04274c0-03a9-4a76-8b37-304b59c8b295" containerName="istio-proxy" Sep 09 16:13:08.852851 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.852835 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:08.856817 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.856793 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bfb318\"/\"kube-root-ca.crt\"" Sep 09 16:13:08.857048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.857033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bfb318\"/\"default-dockercfg-8j55p\"" Sep 09 16:13:08.857235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.857222 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bfb318\"/\"openshift-service-ca.crt\"" Sep 09 16:13:08.857457 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.857441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bfb318\"/\"explicit-model-bfb318-kserve-self-signed-certs\"" Sep 09 16:13:08.858044 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:08.858022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:09.023375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.023587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.023587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65gj\" (UniqueName: \"kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.023587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.023587 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.023751 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.023592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125277 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65gj\" (UniqueName: \"kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125277 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.125749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.126197 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.125987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.126197 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.126047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.126197 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.126108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.127823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.127802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.128198 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.128176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.134065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.134044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65gj\" (UniqueName: \"kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj\") pod \"explicit-model-bfb318-kserve-5675dd5c77-j6cdf\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.163841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.163806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:09.441458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:09.441427 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:09.442741 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:13:09.442712 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88bb7eaa_6c61_42dc_84d7_0323c884e810.slice/crio-5db0825efd9c5d778028860ac62ab2b53d4e864215a7097103334063dc1660e1 WatchSource:0}: Error finding container 5db0825efd9c5d778028860ac62ab2b53d4e864215a7097103334063dc1660e1: Status 404 returned error can't find the container with id 5db0825efd9c5d778028860ac62ab2b53d4e864215a7097103334063dc1660e1 Sep 09 16:13:10.419335 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:10.419293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" event={"ID":"88bb7eaa-6c61-42dc-84d7-0323c884e810","Type":"ContainerStarted","Data":"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298"} Sep 09 16:13:10.419804 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:10.419344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" event={"ID":"88bb7eaa-6c61-42dc-84d7-0323c884e810","Type":"ContainerStarted","Data":"5db0825efd9c5d778028860ac62ab2b53d4e864215a7097103334063dc1660e1"} Sep 09 16:13:10.466816 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:10.466756 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" podStartSLOduration=2.466735488 podStartE2EDuration="2.466735488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:10.466304329 +0000 UTC m=+1309.807496158" watchObservedRunningTime="2026-09-09 16:13:10.466735488 +0000 UTC m=+1309.807927318" Sep 09 16:13:18.747076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:18.747039 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:13:19.164066 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:19.164028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:19.164246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:19.164079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:19.176525 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:19.176492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:19.482458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:19.482352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:25.471646 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.471609 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:13:25.477095 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.477068 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.479979 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.479948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-677907eb-openshift-default-dockercfg-7vnjm\"" Sep 09 16:13:25.488836 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.488807 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:13:25.586224 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.586185 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.586534 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.586501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.586694 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.586677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.586915 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.586897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.587105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.587073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.587244 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.587229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k94st\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.587376 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.587361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.587513 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.587498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.587638 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.587621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.688974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.688936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.688974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.688975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.688994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k94st\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689187 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689538 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.689810 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.689792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.691899 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.691875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.692096 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.692081 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.701726 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.701706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.701896 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.701878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k94st\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st\") pod \"e2e-mt-677907eb-openshift-default-d655f65dd-25qnl\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.790381 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.790292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:25.828704 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:25.828669 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:26.075284 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.075259 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:13:26.076946 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:13:26.076923 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda395be04_c7a0_47c2_9979_4be635ec6686.slice/crio-60ac8373c59f786999c60dda38b3c714f5988ae31bb0797431fdfe7830b09ad7 WatchSource:0}: Error finding container 60ac8373c59f786999c60dda38b3c714f5988ae31bb0797431fdfe7830b09ad7: Status 404 returned error can't find the container with id 60ac8373c59f786999c60dda38b3c714f5988ae31bb0797431fdfe7830b09ad7 Sep 09 16:13:26.079540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.079498 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:26.079665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.079589 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:26.079665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.079629 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:26.512455 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.512394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerStarted","Data":"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195"} Sep 09 16:13:26.512944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.512500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerStarted","Data":"60ac8373c59f786999c60dda38b3c714f5988ae31bb0797431fdfe7830b09ad7"} Sep 09 16:13:26.534504 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.534462 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podStartSLOduration=1.534448413 podStartE2EDuration="1.534448413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:13:26.532753553 +0000 UTC m=+1325.873945383" watchObservedRunningTime="2026-09-09 16:13:26.534448413 +0000 UTC m=+1325.875640242" Sep 09 16:13:26.791289 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:26.791193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:27.164601 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.164562 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq"] Sep 09 16:13:27.168367 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.168351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:27.171462 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.171440 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:13:27.171585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.171441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:13:27.178665 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.178636 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq"] Sep 09 16:13:27.204779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.204747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq\" (UID: \"e44cdcd2-b3d9-4913-a827-4f6180a36609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:27.305864 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.305827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq\" (UID: \"e44cdcd2-b3d9-4913-a827-4f6180a36609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:27.306516 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.306493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq\" (UID: \"e44cdcd2-b3d9-4913-a827-4f6180a36609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:27.481200 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.481104 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:27.760001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.759974 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq"] Sep 09 16:13:27.761531 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:13:27.761505 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44cdcd2_b3d9_4913_a827_4f6180a36609.slice/crio-c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f WatchSource:0}: Error finding container c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f: Status 404 returned error can't find the container with id c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f Sep 09 16:13:27.796033 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:27.796012 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:28.526218 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:28.526182 2562 generic.go:356] "Generic (PLEG): container finished" podID="e44cdcd2-b3d9-4913-a827-4f6180a36609" containerID="a89db23c42dc0e3d7e6853500f9ad935ed7eb8b7bea3933c1ec9801e7f0cfe3b" exitCode=0 Sep 09 16:13:28.526373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:28.526233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" event={"ID":"e44cdcd2-b3d9-4913-a827-4f6180a36609","Type":"ContainerDied","Data":"a89db23c42dc0e3d7e6853500f9ad935ed7eb8b7bea3933c1ec9801e7f0cfe3b"} Sep 09 16:13:28.526373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:28.526295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" event={"ID":"e44cdcd2-b3d9-4913-a827-4f6180a36609","Type":"ContainerStarted","Data":"c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f"} Sep 09 16:13:28.526619 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:28.526599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:28.527501 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:28.527484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:29.674753 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:29.674730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:29.733637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:29.733613 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca\") pod \"e44cdcd2-b3d9-4913-a827-4f6180a36609\" (UID: \"e44cdcd2-b3d9-4913-a827-4f6180a36609\") " Sep 09 16:13:29.733950 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:29.733930 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e44cdcd2-b3d9-4913-a827-4f6180a36609" (UID: "e44cdcd2-b3d9-4913-a827-4f6180a36609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:13:29.835304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:29.835232 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e44cdcd2-b3d9-4913-a827-4f6180a36609-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:30.539235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:30.539198 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" event={"ID":"e44cdcd2-b3d9-4913-a827-4f6180a36609","Type":"ContainerDied","Data":"c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f"} Sep 09 16:13:30.539462 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:30.539256 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16a190c40c0f85457d9647a6bfbd68568de837c82cfea5511baf358f317031f" Sep 09 16:13:30.539462 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:30.539213 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq" Sep 09 16:13:35.338339 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:35.338304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:13:38.259341 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:38.259299 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq"] Sep 09 16:13:38.261775 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:38.261745 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7b8369-qz2hq"] Sep 09 16:13:39.194906 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:39.194867 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44cdcd2-b3d9-4913-a827-4f6180a36609" path="/var/lib/kubelet/pods/e44cdcd2-b3d9-4913-a827-4f6180a36609/volumes" Sep 09 16:13:40.854301 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:40.854258 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" podUID="88bb7eaa-6c61-42dc-84d7-0323c884e810" containerName="main" containerID="cri-o://0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298" gracePeriod=45 Sep 09 16:13:41.157020 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.156995 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:41.257149 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257118 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257149 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257150 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257280 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257354 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257521 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257387 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s65gj\" (UniqueName: \"kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257521 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache\") pod \"88bb7eaa-6c61-42dc-84d7-0323c884e810\" (UID: \"88bb7eaa-6c61-42dc-84d7-0323c884e810\") " Sep 09 16:13:41.257521 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257480 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:41.257681 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257595 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home" (OuterVolumeSpecName: "home") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:41.257787 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257751 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache" (OuterVolumeSpecName: "model-cache") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:41.257914 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257885 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.257914 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257907 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.258034 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.257922 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.259659 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.259627 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj" (OuterVolumeSpecName: "kube-api-access-s65gj") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "kube-api-access-s65gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:13:41.259659 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.259647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:13:41.259802 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.259642 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm" (OuterVolumeSpecName: "dshm") pod "88bb7eaa-6c61-42dc-84d7-0323c884e810" (UID: "88bb7eaa-6c61-42dc-84d7-0323c884e810"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:41.359164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.359128 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb7eaa-6c61-42dc-84d7-0323c884e810-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.359164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.359157 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb7eaa-6c61-42dc-84d7-0323c884e810-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.359164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.359170 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65gj\" (UniqueName: \"kubernetes.io/projected/88bb7eaa-6c61-42dc-84d7-0323c884e810-kube-api-access-s65gj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:41.617108 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.617059 2562 generic.go:356] "Generic (PLEG): container finished" podID="88bb7eaa-6c61-42dc-84d7-0323c884e810" containerID="0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298" exitCode=0 Sep 09 16:13:41.617309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.617153 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" event={"ID":"88bb7eaa-6c61-42dc-84d7-0323c884e810","Type":"ContainerDied","Data":"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298"} Sep 09 16:13:41.617309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.617189 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" event={"ID":"88bb7eaa-6c61-42dc-84d7-0323c884e810","Type":"ContainerDied","Data":"5db0825efd9c5d778028860ac62ab2b53d4e864215a7097103334063dc1660e1"} Sep 09 16:13:41.617309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.617212 2562 scope.go:117] "RemoveContainer" containerID="0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298" Sep 09 16:13:41.617503 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.617372 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf" Sep 09 16:13:41.632090 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.632064 2562 scope.go:117] "RemoveContainer" containerID="0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298" Sep 09 16:13:41.632639 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:13:41.632607 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298\": container with ID starting with 0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298 not found: ID does not exist" containerID="0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298" Sep 09 16:13:41.632734 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.632649 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298"} err="failed to get container status \"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298\": rpc error: code = NotFound desc = could not find container \"0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298\": container with ID starting with 0af0e30a860c5500ee4fddb926bd5fc5a400e2a9f185b2e0d45aa42cfd058298 not found: ID does not exist" Sep 09 16:13:41.643952 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.643924 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:41.650529 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:41.650503 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bfb318/explicit-model-bfb318-kserve-5675dd5c77-j6cdf"] Sep 09 16:13:43.191074 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:43.191039 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88bb7eaa-6c61-42dc-84d7-0323c884e810" path="/var/lib/kubelet/pods/88bb7eaa-6c61-42dc-84d7-0323c884e810/volumes" Sep 09 16:13:43.811428 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:43.811375 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:13:43.811812 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:43.811755 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" containerID="cri-o://07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5" gracePeriod=30 Sep 09 16:13:44.567586 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:44.567545 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:13:44.568008 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:44.567645 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:13:45.364902 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:45.364863 2562 patch_prober.go:28] interesting pod/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:13:45.365075 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:45.364942 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:13:45.401775 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:45.401740 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:13:45.401999 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:45.401974 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" containerID="cri-o://f27864883b0c271883c4a0288f8a1a448766682bffcc040ddd5b8f7df9820440" gracePeriod=30 Sep 09 16:13:46.648941 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.648907 2562 generic.go:356] "Generic (PLEG): container finished" podID="a395be04-c7a0-47c2-9979-4be635ec6686" containerID="2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195" exitCode=0 Sep 09 16:13:46.649348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.648976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerDied","Data":"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195"} Sep 09 16:13:46.649391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.649379 2562 scope.go:117] "RemoveContainer" containerID="2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195" Sep 09 16:13:46.650588 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.650559 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:46.650666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.650641 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:46.650703 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:46.650673 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:13:47.664111 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:47.664067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerStarted","Data":"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9"} Sep 09 16:13:47.791615 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:47.791571 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:47.791833 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:47.791719 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 16:13:47.791833 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:47.791764 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 16:13:48.333855 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:48.333818 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:13:48.457193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:48.457157 2562 patch_prober.go:28] interesting pod/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:13:48.457376 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:48.457220 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:13:48.793846 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:48.793806 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 16:13:48.794317 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:48.793884 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 16:13:49.116943 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.116919 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:13:49.139969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.139937 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjkr2\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.139975 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140005 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140130 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140209 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140246 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket\") pod \"1486de15-9f13-4fbd-bb7a-d212f72a891c\" (UID: \"1486de15-9f13-4fbd-bb7a-d212f72a891c\") " Sep 09 16:13:49.140555 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140280 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:49.140555 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140397 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:49.140679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140650 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:49.140679 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140665 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.140774 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140686 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.140774 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140747 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:13:49.140872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.140586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:49.142463 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.142314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2" (OuterVolumeSpecName: "kube-api-access-jjkr2") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "kube-api-access-jjkr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:13:49.142624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.142494 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:13:49.142990 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.142966 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:13:49.143279 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.143243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1486de15-9f13-4fbd-bb7a-d212f72a891c" (UID: "1486de15-9f13-4fbd-bb7a-d212f72a891c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:49.241666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241638 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241662 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1486de15-9f13-4fbd-bb7a-d212f72a891c-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241666 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241672 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241680 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241687 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1486de15-9f13-4fbd-bb7a-d212f72a891c-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241697 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkr2\" (UniqueName: \"kubernetes.io/projected/1486de15-9f13-4fbd-bb7a-d212f72a891c-kube-api-access-jjkr2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.241844 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.241706 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1486de15-9f13-4fbd-bb7a-d212f72a891c-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:49.676954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.676920 2562 generic.go:356] "Generic (PLEG): container finished" podID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerID="07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5" exitCode=0 Sep 09 16:13:49.677161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.676966 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" Sep 09 16:13:49.677161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.677011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerDied","Data":"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5"} Sep 09 16:13:49.677161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.677074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx" event={"ID":"1486de15-9f13-4fbd-bb7a-d212f72a891c","Type":"ContainerDied","Data":"cb87a53383fcaef64ca88d0b353a96e2b2f37b4df8c604c1ed45c70fba99cd8c"} Sep 09 16:13:49.677161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.677104 2562 scope.go:117] "RemoveContainer" containerID="07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5" Sep 09 16:13:49.690988 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.689791 2562 scope.go:117] "RemoveContainer" containerID="2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713" Sep 09 16:13:49.708513 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.708482 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:13:49.709196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.708769 2562 scope.go:117] "RemoveContainer" containerID="07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5" Sep 09 16:13:49.709196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.709085 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-bfb318-openshift-default-6bdffbd6d-p9ghx"] Sep 09 16:13:49.709694 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:13:49.709661 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5\": container with ID starting with 07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5 not found: ID does not exist" containerID="07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5" Sep 09 16:13:49.709796 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.709702 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5"} err="failed to get container status \"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5\": rpc error: code = NotFound desc = could not find container \"07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5\": container with ID starting with 07bf0a204fbea46bc61c04c728eab6d4956f29065d6ac29c5de9c32fe88f0eb5 not found: ID does not exist" Sep 09 16:13:49.709796 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.709728 2562 scope.go:117] "RemoveContainer" containerID="2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713" Sep 09 16:13:49.710040 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:13:49.710009 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713\": container with ID starting with 2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713 not found: ID does not exist" containerID="2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713" Sep 09 16:13:49.710127 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.710045 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713"} err="failed to get container status \"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713\": rpc error: code = NotFound desc = could not find container \"2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713\": container with ID starting with 2d48e75a2acf7d8cd48d129c99478c4390b66dd447c4b39327b92d867c810713 not found: ID does not exist" Sep 09 16:13:49.792815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.792780 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 16:13:49.793064 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:49.793033 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 16:13:50.700658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.698383 2562 generic.go:356] "Generic (PLEG): container finished" podID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerID="f27864883b0c271883c4a0288f8a1a448766682bffcc040ddd5b8f7df9820440" exitCode=0 Sep 09 16:13:50.700658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.698552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerDied","Data":"f27864883b0c271883c4a0288f8a1a448766682bffcc040ddd5b8f7df9820440"} Sep 09 16:13:50.700658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.698606 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" event={"ID":"8fd744b4-2594-4359-bff6-07aa0c8d5bec","Type":"ContainerDied","Data":"0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049"} Sep 09 16:13:50.700658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.698632 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc6a471714da67209a5f1d39ac107687a8d425fbaef14f1d67251459d69c049" Sep 09 16:13:50.700658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.698650 2562 scope.go:117] "RemoveContainer" containerID="9de04cccec4ad46f2e163322e1693128030ede13ede4c821442881cb5f0816fc" Sep 09 16:13:50.708711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.705589 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:13:50.763850 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.763826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.763976 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.763862 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.763976 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.763896 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.763976 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.763941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nch5n\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.763991 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764033 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764083 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764101 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:50.764193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764161 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs\") pod \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\" (UID: \"8fd744b4-2594-4359-bff6-07aa0c8d5bec\") " Sep 09 16:13:50.764424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:50.764424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764339 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:13:50.764424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:50.764596 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:50.764697 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764677 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.764768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764701 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.764768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764714 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.764768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764728 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.764768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.764741 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.766101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.766075 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:13:50.766208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.766138 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:13:50.766208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.766150 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:13:50.766291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.766216 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n" (OuterVolumeSpecName: "kube-api-access-nch5n") pod "8fd744b4-2594-4359-bff6-07aa0c8d5bec" (UID: "8fd744b4-2594-4359-bff6-07aa0c8d5bec"). InnerVolumeSpecName "kube-api-access-nch5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:13:50.797435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.793740 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 16:13:50.797435 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.793803 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 16:13:50.865840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.865812 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nch5n\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-kube-api-access-nch5n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.865840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.865838 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.865840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.865847 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:50.866071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:50.865857 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd744b4-2594-4359-bff6-07aa0c8d5bec-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:13:51.199631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.199587 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" path="/var/lib/kubelet/pods/1486de15-9f13-4fbd-bb7a-d212f72a891c/volumes" Sep 09 16:13:51.709918 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.709889 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd" Sep 09 16:13:51.746882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.746856 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:13:51.752806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.752769 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7b8369-openshift-default-c85cbf44d-hlpfd"] Sep 09 16:13:51.794390 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.794349 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 16:13:51.794559 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:51.794446 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 16:13:53.006663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:53.006614 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:53.007745 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:53.007720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:53.009535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:53.009511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:13:53.190213 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:53.190181 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" path="/var/lib/kubelet/pods/8fd744b4-2594-4359-bff6-07aa0c8d5bec/volumes" Sep 09 16:13:58.626032 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:13:58.625988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:14:00.919112 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919071 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld"] Sep 09 16:14:00.919760 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919738 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919762 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919776 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919783 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919806 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bb7eaa-6c61-42dc-84d7-0323c884e810" containerName="main" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919815 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bb7eaa-6c61-42dc-84d7-0323c884e810" containerName="main" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919827 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44cdcd2-b3d9-4913-a827-4f6180a36609" containerName="revoke-keys" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919834 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44cdcd2-b3d9-4913-a827-4f6180a36609" containerName="revoke-keys" Sep 09 16:14:00.919858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919857 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919865 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919969 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919982 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44cdcd2-b3d9-4913-a827-4f6180a36609" containerName="revoke-keys" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.919993 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.920002 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd744b4-2594-4359-bff6-07aa0c8d5bec" containerName="istio-proxy" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.920014 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bb7eaa-6c61-42dc-84d7-0323c884e810" containerName="main" Sep 09 16:14:00.920245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.920022 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:00.924915 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.924896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:00.927822 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.927795 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:14:00.928177 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.927797 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:14:00.929254 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.929232 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld"] Sep 09 16:14:00.962299 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:00.962273 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-677907eb-426ld\" (UID: \"5b98b85a-db86-4337-9c12-ba70289ea4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:01.063083 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.063053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-677907eb-426ld\" (UID: \"5b98b85a-db86-4337-9c12-ba70289ea4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:01.063683 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.063662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-677907eb-426ld\" (UID: \"5b98b85a-db86-4337-9c12-ba70289ea4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:01.237742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.237662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:01.508972 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.508945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld"] Sep 09 16:14:01.511366 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:14:01.511325 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b98b85a_db86_4337_9c12_ba70289ea4aa.slice/crio-dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db WatchSource:0}: Error finding container dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db: Status 404 returned error can't find the container with id dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db Sep 09 16:14:01.776913 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.776824 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" event={"ID":"5b98b85a-db86-4337-9c12-ba70289ea4aa","Type":"ContainerStarted","Data":"9368336cb92ee557c920f475f4c59f8a4d1000c837dc197b6c4a3ea4a159d70e"} Sep 09 16:14:01.776913 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.776874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" event={"ID":"5b98b85a-db86-4337-9c12-ba70289ea4aa","Type":"ContainerStarted","Data":"dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db"} Sep 09 16:14:01.793352 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:01.793311 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" podStartSLOduration=1.793295023 podStartE2EDuration="1.793295023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:01.791107351 +0000 UTC m=+1361.132299182" watchObservedRunningTime="2026-09-09 16:14:01.793295023 +0000 UTC m=+1361.134486853" Sep 09 16:14:02.785470 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:02.785430 2562 generic.go:356] "Generic (PLEG): container finished" podID="5b98b85a-db86-4337-9c12-ba70289ea4aa" containerID="9368336cb92ee557c920f475f4c59f8a4d1000c837dc197b6c4a3ea4a159d70e" exitCode=0 Sep 09 16:14:02.785837 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:02.785502 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" event={"ID":"5b98b85a-db86-4337-9c12-ba70289ea4aa","Type":"ContainerDied","Data":"9368336cb92ee557c920f475f4c59f8a4d1000c837dc197b6c4a3ea4a159d70e"} Sep 09 16:14:03.082973 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:03.082889 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:14:03.083430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:03.083375 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" containerID="cri-o://ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d" gracePeriod=30 Sep 09 16:14:03.925534 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:03.925511 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:03.990528 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:03.990498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca\") pod \"5b98b85a-db86-4337-9c12-ba70289ea4aa\" (UID: \"5b98b85a-db86-4337-9c12-ba70289ea4aa\") " Sep 09 16:14:03.990836 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:03.990814 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b98b85a-db86-4337-9c12-ba70289ea4aa" (UID: "5b98b85a-db86-4337-9c12-ba70289ea4aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:14:04.043565 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.043486 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:14:04.043970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.043956 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:04.044030 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.043971 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1486de15-9f13-4fbd-bb7a-d212f72a891c" containerName="istio-proxy" Sep 09 16:14:04.044030 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.043984 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b98b85a-db86-4337-9c12-ba70289ea4aa" containerName="revoke-keys" Sep 09 16:14:04.044030 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.043990 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b98b85a-db86-4337-9c12-ba70289ea4aa" containerName="revoke-keys" Sep 09 16:14:04.044136 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.044071 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b98b85a-db86-4337-9c12-ba70289ea4aa" containerName="revoke-keys" Sep 09 16:14:04.051584 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.051563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.054507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.054441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-131689-openshift-default-dockercfg-lt5nw\"" Sep 09 16:14:04.061096 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.061072 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:14:04.092091 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092156 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbcz\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092382 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.092537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.092461 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b98b85a-db86-4337-9c12-ba70289ea4aa-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:04.193168 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193194 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbcz\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193325 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.193946 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.193853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.194110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.194091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.195648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.195627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.195768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.195749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.201313 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.201286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.201448 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.201344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbcz\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz\") pod \"e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.365953 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.365923 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:04.642437 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.642394 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:14:04.644263 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:14:04.644236 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d24225_9712_457c_ae5a_252aad07dff1.slice/crio-bdbcc4f93131d8979344c69e76a107d6a2a5b1ecad574b8b61c18b4551c296dc WatchSource:0}: Error finding container bdbcc4f93131d8979344c69e76a107d6a2a5b1ecad574b8b61c18b4551c296dc: Status 404 returned error can't find the container with id bdbcc4f93131d8979344c69e76a107d6a2a5b1ecad574b8b61c18b4551c296dc Sep 09 16:14:04.646430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.646382 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:04.646508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.646464 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:04.646508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.646495 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:04.798998 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.798962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerStarted","Data":"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c"} Sep 09 16:14:04.799181 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.799019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerStarted","Data":"bdbcc4f93131d8979344c69e76a107d6a2a5b1ecad574b8b61c18b4551c296dc"} Sep 09 16:14:04.800464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.800436 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" event={"ID":"5b98b85a-db86-4337-9c12-ba70289ea4aa","Type":"ContainerDied","Data":"dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db"} Sep 09 16:14:04.800464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.800455 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld" Sep 09 16:14:04.800610 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.800484 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc35abb1079dc56d3ee7b1a6d817f1ec03aaebff66c4d0f27aa23e6bae1a58db" Sep 09 16:14:04.819028 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:04.818981 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podStartSLOduration=0.818967128 podStartE2EDuration="818.967128ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:04.817675768 +0000 UTC m=+1364.158867616" watchObservedRunningTime="2026-09-09 16:14:04.818967128 +0000 UTC m=+1364.160158958" Sep 09 16:14:05.366703 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:05.366667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:05.368223 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:05.368201 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:05.368269 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:05.368252 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:06.370356 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:06.370322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:06.812475 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:06.812374 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:06.813223 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:06.813203 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:07.878366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:07.878326 2562 patch_prober.go:28] interesting pod/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:14:07.878757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:07.878384 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:14:08.344197 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.344175 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:14:08.433678 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433651 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.433824 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433711 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.433824 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433753 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps5mq\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.433824 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433799 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.433934 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433918 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.433970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433940 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:08.434003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.433967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.434034 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434022 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.434077 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434054 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:08.434131 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434080 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.434188 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434126 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy\") pod \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\" (UID: \"e8c88c80-ba00-43c7-bb0b-93b5c1529399\") " Sep 09 16:14:08.434277 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434125 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:08.434379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434356 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:08.434379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434361 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:14:08.434603 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434578 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.434699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434606 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.434699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434620 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.434699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434635 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.434699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.434650 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.436080 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.436058 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq" (OuterVolumeSpecName: "kube-api-access-ps5mq") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "kube-api-access-ps5mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:14:08.436080 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.436065 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:08.436201 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.436081 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:14:08.436377 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.436357 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8c88c80-ba00-43c7-bb0b-93b5c1529399" (UID: "e8c88c80-ba00-43c7-bb0b-93b5c1529399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:14:08.535447 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.535358 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.535447 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.535385 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps5mq\" (UniqueName: \"kubernetes.io/projected/e8c88c80-ba00-43c7-bb0b-93b5c1529399-kube-api-access-ps5mq\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.535447 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.535394 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.535447 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.535435 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c88c80-ba00-43c7-bb0b-93b5c1529399-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:08.826221 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.826130 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerID="ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d" exitCode=0 Sep 09 16:14:08.826221 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.826183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" Sep 09 16:14:08.826482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.826219 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerDied","Data":"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d"} Sep 09 16:14:08.826482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.826287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p" event={"ID":"e8c88c80-ba00-43c7-bb0b-93b5c1529399","Type":"ContainerDied","Data":"55c06c5c1ecb512bd2bd53f35ad32afc77c502bb9131aa2e3ea1b93b2e9d757e"} Sep 09 16:14:08.826482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.826316 2562 scope.go:117] "RemoveContainer" containerID="ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d" Sep 09 16:14:08.835969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.835954 2562 scope.go:117] "RemoveContainer" containerID="b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324" Sep 09 16:14:08.844563 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.844541 2562 scope.go:117] "RemoveContainer" containerID="ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d" Sep 09 16:14:08.844820 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:14:08.844804 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d\": container with ID starting with ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d not found: ID does not exist" containerID="ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d" Sep 09 16:14:08.844887 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.844828 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d"} err="failed to get container status \"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d\": rpc error: code = NotFound desc = could not find container \"ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d\": container with ID starting with ff3e38398649df7360c731bfbbc7c417adba10135e45cee0ad7cac86a84fb10d not found: ID does not exist" Sep 09 16:14:08.844887 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.844843 2562 scope.go:117] "RemoveContainer" containerID="b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324" Sep 09 16:14:08.845079 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:14:08.845061 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324\": container with ID starting with b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324 not found: ID does not exist" containerID="b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324" Sep 09 16:14:08.845117 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.845085 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324"} err="failed to get container status \"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324\": rpc error: code = NotFound desc = could not find container \"b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324\": container with ID starting with b9421bc5de8420f6cc75051ed91ef85f8979663ba7cae557369dc6ba6c0be324 not found: ID does not exist" Sep 09 16:14:08.852792 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.852772 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:14:08.859355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:08.859336 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c23ff1-openshift-default-6b678d6958-j527p"] Sep 09 16:14:09.194985 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:09.194942 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" path="/var/lib/kubelet/pods/e8c88c80-ba00-43c7-bb0b-93b5c1529399/volumes" Sep 09 16:14:11.985396 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:11.985359 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld"] Sep 09 16:14:11.987251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:11.987229 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-677907eb-426ld"] Sep 09 16:14:12.648286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:12.648256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:14:13.189490 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:13.189459 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b98b85a-db86-4337-9c12-ba70289ea4aa" path="/var/lib/kubelet/pods/5b98b85a-db86-4337-9c12-ba70289ea4aa/volumes" Sep 09 16:14:18.447708 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:18.447675 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:14:18.448183 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:18.447870 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" containerID="cri-o://5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9" gracePeriod=30 Sep 09 16:14:21.030662 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:21.030629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:14:22.267612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.266462 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:36150->10.132.0.59:15021: read: connection reset by peer" start-of-body= Sep 09 16:14:22.267612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.266617 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:36150->10.132.0.59:15021: read: connection reset by peer" Sep 09 16:14:22.923691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.923655 2562 generic.go:356] "Generic (PLEG): container finished" podID="77d24225-9712-457c-ae5a-252aad07dff1" containerID="5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c" exitCode=0 Sep 09 16:14:22.923874 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.923727 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerDied","Data":"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c"} Sep 09 16:14:22.924147 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.924130 2562 scope.go:117] "RemoveContainer" containerID="5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c" Sep 09 16:14:22.925244 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.925217 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:22.925310 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.925295 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:22.925350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:22.925332 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:23.010322 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.010294 2562 patch_prober.go:28] interesting pod/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:14:23.010441 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.010343 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:14:23.705230 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.705207 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:14:23.890788 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.890991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.890991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890853 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.890991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k94st\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.890991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890927 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.890991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.890991 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.891286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.891038 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.891286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.891077 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.891286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.891101 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert\") pod \"a395be04-c7a0-47c2-9979-4be635ec6686\" (UID: \"a395be04-c7a0-47c2-9979-4be635ec6686\") " Sep 09 16:14:23.891814 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.891784 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:14:23.892001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.891978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:23.893305 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.892512 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data" (OuterVolumeSpecName: "istio-data") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:23.893305 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.892928 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:23.893305 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.893064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:23.896593 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.896560 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token" (OuterVolumeSpecName: "istio-token") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:14:23.896764 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.896487 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:14:23.897580 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.897543 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:14:23.899332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.899302 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st" (OuterVolumeSpecName: "kube-api-access-k94st") pod "a395be04-c7a0-47c2-9979-4be635ec6686" (UID: "a395be04-c7a0-47c2-9979-4be635ec6686"). InnerVolumeSpecName "kube-api-access-k94st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:14:23.931764 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.931734 2562 generic.go:356] "Generic (PLEG): container finished" podID="a395be04-c7a0-47c2-9979-4be635ec6686" containerID="5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9" exitCode=0 Sep 09 16:14:23.931905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.931785 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" Sep 09 16:14:23.931905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.931816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerDied","Data":"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9"} Sep 09 16:14:23.931905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.931885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl" event={"ID":"a395be04-c7a0-47c2-9979-4be635ec6686","Type":"ContainerDied","Data":"60ac8373c59f786999c60dda38b3c714f5988ae31bb0797431fdfe7830b09ad7"} Sep 09 16:14:23.931905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.931902 2562 scope.go:117] "RemoveContainer" containerID="5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9" Sep 09 16:14:23.933917 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.933896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerStarted","Data":"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb"} Sep 09 16:14:23.941588 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.941570 2562 scope.go:117] "RemoveContainer" containerID="2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195" Sep 09 16:14:23.949549 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.949533 2562 scope.go:117] "RemoveContainer" containerID="5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9" Sep 09 16:14:23.949793 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:14:23.949773 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9\": container with ID starting with 5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9 not found: ID does not exist" containerID="5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9" Sep 09 16:14:23.949852 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.949800 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9"} err="failed to get container status \"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9\": rpc error: code = NotFound desc = could not find container \"5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9\": container with ID starting with 5f7770cc02c47b3b1237aa1bf946bd52511f4c6ece77c0a617db32d5166290c9 not found: ID does not exist" Sep 09 16:14:23.949852 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.949815 2562 scope.go:117] "RemoveContainer" containerID="2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195" Sep 09 16:14:23.950030 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:14:23.950014 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195\": container with ID starting with 2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195 not found: ID does not exist" containerID="2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195" Sep 09 16:14:23.950070 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.950033 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195"} err="failed to get container status \"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195\": rpc error: code = NotFound desc = could not find container \"2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195\": container with ID starting with 2390ce4848c4fc97dcda3d4030945b7f03e916ee2fcd6fbd678b7f5929e8f195 not found: ID does not exist" Sep 09 16:14:23.967866 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.967846 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:14:23.970975 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.970954 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-677907eb-openshift-default-d655f65dd-25qnl"] Sep 09 16:14:23.992657 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992614 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992657 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992656 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992676 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992690 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k94st\" (UniqueName: \"kubernetes.io/projected/a395be04-c7a0-47c2-9979-4be635ec6686-kube-api-access-k94st\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992705 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992718 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395be04-c7a0-47c2-9979-4be635ec6686-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992730 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992744 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395be04-c7a0-47c2-9979-4be635ec6686-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:23.992907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:23.992756 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395be04-c7a0-47c2-9979-4be635ec6686-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:14:24.366331 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:24.366300 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:24.366541 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:24.366519 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:24.366600 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:24.366567 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:25.195241 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:25.195204 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" path="/var/lib/kubelet/pods/a395be04-c7a0-47c2-9979-4be635ec6686/volumes" Sep 09 16:14:25.367154 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:25.367119 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:25.367318 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:25.367175 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:26.366783 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:26.366744 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:26.367148 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:26.366802 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:27.366903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:27.366863 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:27.367303 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:27.366922 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:28.366306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:28.366263 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:28.366504 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:28.366321 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:29.366375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:29.366339 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 09 16:14:29.366775 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:29.366391 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 09 16:14:30.369515 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:30.369484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:30.369985 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:30.369873 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:30.370562 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:30.370540 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:14:32.370646 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.370606 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:14:32.371069 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371055 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371069 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371070 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371078 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371084 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371090 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371096 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371109 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.371152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371115 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.371363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371179 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371187 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.371363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371196 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c88c80-ba00-43c7-bb0b-93b5c1529399" containerName="istio-proxy" Sep 09 16:14:32.371363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.371203 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a395be04-c7a0-47c2-9979-4be635ec6686" containerName="istio-proxy" Sep 09 16:14:32.376152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.376126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.378645 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.378625 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-13d59f-openshift-default-dockercfg-w6v9p\"" Sep 09 16:14:32.386083 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.386058 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:14:32.483218 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483369 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483369 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483369 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgddd\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.483556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.483539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgddd\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.584925 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.585249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.584965 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.585366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.585341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.585670 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.585643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.585914 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.585892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.587379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.587348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.587865 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.587843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.592511 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.592484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.592605 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.592530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgddd\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd\") pod \"e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.688971 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.688884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:32.963458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.963432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:14:32.964907 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:14:32.964877 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c26e17_7004_40e3_8d24_30437b9b077d.slice/crio-69df6d773633de4867672b286820328fb83150731a745f7a14540d3b5fba509f WatchSource:0}: Error finding container 69df6d773633de4867672b286820328fb83150731a745f7a14540d3b5fba509f: Status 404 returned error can't find the container with id 69df6d773633de4867672b286820328fb83150731a745f7a14540d3b5fba509f Sep 09 16:14:32.967153 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.967120 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:32.967264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.967201 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:32.967264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.967243 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:32.986304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:32.986276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerStarted","Data":"69df6d773633de4867672b286820328fb83150731a745f7a14540d3b5fba509f"} Sep 09 16:14:33.993005 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:33.992968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerStarted","Data":"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d"} Sep 09 16:14:34.012779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:34.012729 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podStartSLOduration=2.012714947 podStartE2EDuration="2.012714947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:34.011632817 +0000 UTC m=+1393.352824646" watchObservedRunningTime="2026-09-09 16:14:34.012714947 +0000 UTC m=+1393.353906776" Sep 09 16:14:34.689049 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:34.689005 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:34.694074 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:34.694047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:34.999291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:34.999211 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:35.000291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:35.000272 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:42.229444 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:42.229391 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:14:51.041560 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:51.041519 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:14:51.042057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:51.041597 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:14:52.101072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.101040 2562 generic.go:356] "Generic (PLEG): container finished" podID="16c26e17-7004-40e3-8d24-30437b9b077d" containerID="83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d" exitCode=0 Sep 09 16:14:52.101510 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.101110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerDied","Data":"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d"} Sep 09 16:14:52.101563 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.101517 2562 scope.go:117] "RemoveContainer" containerID="83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d" Sep 09 16:14:52.102617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.102585 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:52.102724 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.102679 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:52.102724 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:52.102709 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:14:53.109829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:53.109791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerStarted","Data":"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9"} Sep 09 16:14:53.690163 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:53.690129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:53.690579 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:53.690483 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 09 16:14:53.690753 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:53.690731 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 09 16:14:54.689993 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:54.689953 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 09 16:14:54.690496 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:54.690025 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 09 16:14:55.705457 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:55.697504 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 09 16:14:55.705457 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:55.697582 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 09 16:14:56.689517 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:56.689474 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 09 16:14:56.689826 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:56.689795 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 09 16:14:57.689951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:57.689913 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 09 16:14:57.690459 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:57.689982 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 09 16:14:58.693354 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:58.693322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:58.693807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:58.693633 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:14:58.694380 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:14:58.694364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:15:00.144740 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.144709 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5"] Sep 09 16:15:00.149117 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.149095 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx"] Sep 09 16:15:00.149241 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.149212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" Sep 09 16:15:00.152068 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.152042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:15:00.153749 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.153730 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" Sep 09 16:15:00.154490 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.154469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5"] Sep 09 16:15:00.162651 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.162635 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" Sep 09 16:15:00.170235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.170138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" Sep 09 16:15:00.170624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.170600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx"] Sep 09 16:15:00.458721 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.458694 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5"] Sep 09 16:15:00.460220 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:15:00.460189 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1ef53e_42e1_42d2_8cdb_e322f46b51f3.slice/crio-b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4 WatchSource:0}: Error finding container b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4: Status 404 returned error can't find the container with id b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4 Sep 09 16:15:00.475832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:00.475808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx"] Sep 09 16:15:00.477012 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:15:00.476990 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064ff2f7_fea2_42d8_ae69_0fe4a576e4e9.slice/crio-b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095 WatchSource:0}: Error finding container b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095: Status 404 returned error can't find the container with id b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095 Sep 09 16:15:01.158281 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.158246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" event={"ID":"064ff2f7-fea2-42d8-ae69-0fe4a576e4e9","Type":"ContainerStarted","Data":"8dd6b765328266f4fdb2303d542da6f0a05787eef494252b1f4c6745e4daac48"} Sep 09 16:15:01.158690 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.158308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" event={"ID":"064ff2f7-fea2-42d8-ae69-0fe4a576e4e9","Type":"ContainerStarted","Data":"b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095"} Sep 09 16:15:01.159857 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.159833 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" event={"ID":"fb1ef53e-42e1-42d2-8cdb-e322f46b51f3","Type":"ContainerStarted","Data":"1aa96d4726c875173fe5125b83fff227e6d105a45de5afec8ad6307a68e20343"} Sep 09 16:15:01.159950 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.159876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" event={"ID":"fb1ef53e-42e1-42d2-8cdb-e322f46b51f3","Type":"ContainerStarted","Data":"b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4"} Sep 09 16:15:01.175458 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.175400 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" podStartSLOduration=1.175385986 podStartE2EDuration="1.175385986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:01.173882244 +0000 UTC m=+1420.515074074" watchObservedRunningTime="2026-09-09 16:15:01.175385986 +0000 UTC m=+1420.516577816" Sep 09 16:15:01.190272 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.190224 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" podStartSLOduration=1.190207634 podStartE2EDuration="1.190207634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:01.189319594 +0000 UTC m=+1420.530511424" watchObservedRunningTime="2026-09-09 16:15:01.190207634 +0000 UTC m=+1420.531399460" Sep 09 16:15:01.611111 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.611076 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:15:01.615703 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.615684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.618603 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.618583 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-008161-openshift-default-dockercfg-t2l99\"" Sep 09 16:15:01.626501 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.626475 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:15:01.790930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.790844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.790930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.790897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trq6b\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.790982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791296 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.791296 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.791175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892495 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892495 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trq6b\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.892742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893081 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.892746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893081 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.893036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893185 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.893083 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893185 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.893122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893185 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.893173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.893464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.893447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.895084 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.895064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.895209 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.895189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.909307 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.909284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trq6b\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.910353 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.910334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token\") pod \"e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:01.927880 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:01.927861 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:02.167631 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.167597 2562 generic.go:356] "Generic (PLEG): container finished" podID="fb1ef53e-42e1-42d2-8cdb-e322f46b51f3" containerID="1aa96d4726c875173fe5125b83fff227e6d105a45de5afec8ad6307a68e20343" exitCode=0 Sep 09 16:15:02.168050 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.167686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" event={"ID":"fb1ef53e-42e1-42d2-8cdb-e322f46b51f3","Type":"ContainerDied","Data":"1aa96d4726c875173fe5125b83fff227e6d105a45de5afec8ad6307a68e20343"} Sep 09 16:15:02.169299 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.169274 2562 generic.go:356] "Generic (PLEG): container finished" podID="064ff2f7-fea2-42d8-ae69-0fe4a576e4e9" containerID="8dd6b765328266f4fdb2303d542da6f0a05787eef494252b1f4c6745e4daac48" exitCode=0 Sep 09 16:15:02.169451 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.169358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" event={"ID":"064ff2f7-fea2-42d8-ae69-0fe4a576e4e9","Type":"ContainerDied","Data":"8dd6b765328266f4fdb2303d542da6f0a05787eef494252b1f4c6745e4daac48"} Sep 09 16:15:02.207216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.207192 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:15:02.210509 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:15:02.210483 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe613ed_90ca_4586_bcd5_f84311a88788.slice/crio-26129a4dca807f8d1e235899948bc5f5b3ba3e5b7cabac480ef3519153ea04ff WatchSource:0}: Error finding container 26129a4dca807f8d1e235899948bc5f5b3ba3e5b7cabac480ef3519153ea04ff: Status 404 returned error can't find the container with id 26129a4dca807f8d1e235899948bc5f5b3ba3e5b7cabac480ef3519153ea04ff Sep 09 16:15:02.213014 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.212974 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:02.213093 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.213061 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:02.213137 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:02.213103 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:03.176157 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.176111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerStarted","Data":"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1"} Sep 09 16:15:03.176574 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.176179 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerStarted","Data":"26129a4dca807f8d1e235899948bc5f5b3ba3e5b7cabac480ef3519153ea04ff"} Sep 09 16:15:03.203123 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.203064 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podStartSLOduration=2.203041045 podStartE2EDuration="2.203041045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:03.198783135 +0000 UTC m=+1422.539974967" watchObservedRunningTime="2026-09-09 16:15:03.203041045 +0000 UTC m=+1422.544232876" Sep 09 16:15:03.331653 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.331626 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" Sep 09 16:15:03.355459 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.355403 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" Sep 09 16:15:03.928702 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.928668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:03.933755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:03.933730 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:04.184588 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.184504 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" Sep 09 16:15:04.184588 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.184552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5" event={"ID":"fb1ef53e-42e1-42d2-8cdb-e322f46b51f3","Type":"ContainerDied","Data":"b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4"} Sep 09 16:15:04.185065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.184601 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3aa4eeebe63119f75c6f745fdba935b516e634824bd9a201c8a69d4d2d018b4" Sep 09 16:15:04.186006 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.185981 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" Sep 09 16:15:04.186006 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.185986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx" event={"ID":"064ff2f7-fea2-42d8-ae69-0fe4a576e4e9","Type":"ContainerDied","Data":"b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095"} Sep 09 16:15:04.186204 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.186042 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d540f01110d367e84ae57c2b3aaef83758dd535139b63d874261cd533f8095" Sep 09 16:15:04.186362 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.186336 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:04.187239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:04.187221 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:12.239156 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:12.239117 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:15:19.139847 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.139806 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:15:19.140548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140526 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1ef53e-42e1-42d2-8cdb-e322f46b51f3" containerName="cleanup" Sep 09 16:15:19.140650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140551 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1ef53e-42e1-42d2-8cdb-e322f46b51f3" containerName="cleanup" Sep 09 16:15:19.140650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140611 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064ff2f7-fea2-42d8-ae69-0fe4a576e4e9" containerName="cleanup" Sep 09 16:15:19.140650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140627 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="064ff2f7-fea2-42d8-ae69-0fe4a576e4e9" containerName="cleanup" Sep 09 16:15:19.140819 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140732 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1ef53e-42e1-42d2-8cdb-e322f46b51f3" containerName="cleanup" Sep 09 16:15:19.140819 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.140749 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="064ff2f7-fea2-42d8-ae69-0fe4a576e4e9" containerName="cleanup" Sep 09 16:15:19.144720 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.144699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.147601 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.147574 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"default-dockercfg-m6knh\"" Sep 09 16:15:19.147601 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.147586 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"kube-root-ca.crt\"" Sep 09 16:15:19.147773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.147598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"openshift-service-ca.crt\"" Sep 09 16:15:19.147773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.147574 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"auth-test-model-131689-kserve-self-signed-certs\"" Sep 09 16:15:19.152366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.152341 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:15:19.258026 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.257991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.258026 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.258029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.258293 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.258157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blb6n\" (UniqueName: \"kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.258293 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.258204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.258293 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.258253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.258293 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.258293 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359153 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359314 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359314 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blb6n\" (UniqueName: \"kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359694 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359659 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.359742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.359724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.361426 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.361390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.361656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.361637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.368003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.367978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blb6n\" (UniqueName: \"kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n\") pod \"auth-test-model-131689-kserve-6ff7577c9-w5htl\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.457609 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.457513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:19.744281 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:19.744252 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:15:19.746948 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:15:19.746919 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod590c553e_17c6_4db7_b937_d463e04bd6b6.slice/crio-1f2cadf18f207689353b9ca1404ba1b391362d6cab596d572de7bb23362aa277 WatchSource:0}: Error finding container 1f2cadf18f207689353b9ca1404ba1b391362d6cab596d572de7bb23362aa277: Status 404 returned error can't find the container with id 1f2cadf18f207689353b9ca1404ba1b391362d6cab596d572de7bb23362aa277 Sep 09 16:15:20.227623 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:20.227584 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 16:15:20.228088 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:20.227657 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 16:15:20.279834 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:20.279799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" event={"ID":"590c553e-17c6-4db7-b937-d463e04bd6b6","Type":"ContainerStarted","Data":"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f"} Sep 09 16:15:20.280029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:20.279843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" event={"ID":"590c553e-17c6-4db7-b937-d463e04bd6b6","Type":"ContainerStarted","Data":"1f2cadf18f207689353b9ca1404ba1b391362d6cab596d572de7bb23362aa277"} Sep 09 16:15:20.297641 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:20.297594 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" podStartSLOduration=1.297578685 podStartE2EDuration="1.297578685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:20.296122767 +0000 UTC m=+1439.637314611" watchObservedRunningTime="2026-09-09 16:15:20.297578685 +0000 UTC m=+1439.638770513" Sep 09 16:15:21.939013 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:21.938942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:15:22.295721 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.295688 2562 generic.go:356] "Generic (PLEG): container finished" podID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerID="6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1" exitCode=0 Sep 09 16:15:22.295896 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.295765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerDied","Data":"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1"} Sep 09 16:15:22.296150 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.296136 2562 scope.go:117] "RemoveContainer" containerID="6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1" Sep 09 16:15:22.297193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.297162 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:22.297328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.297229 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:22.297328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:22.297262 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:23.304066 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:23.304037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerStarted","Data":"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938"} Sep 09 16:15:23.928649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:23.928609 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:23.928936 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:23.928852 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:23.928936 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:23.928911 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:24.929092 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:24.929054 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:24.929523 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:24.929114 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:25.928994 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:25.928957 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:25.929164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:25.929018 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:26.928960 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:26.928924 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:26.929135 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:26.928977 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:27.929105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:27.929064 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:27.929509 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:27.929124 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:28.928948 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:28.928912 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 09 16:15:28.929126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:28.928970 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 09 16:15:29.458651 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.458616 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:29.458811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.458671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:29.471264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.471242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:29.931621 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.931589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:29.932016 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.931872 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:29.932624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:29.932599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:15:30.359385 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:30.359360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:15:39.955443 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:39.955386 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:15:39.960389 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:39.960371 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:39.962983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:39.962957 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4c4ad334-openshift-default-dockercfg-dxsn2\"" Sep 09 16:15:39.969811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:39.969790 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:15:40.064848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.064808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.064848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.064863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.064895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.064920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.065031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065115 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.065078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnwg\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.065142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.065180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.065346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.065204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166779 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166977 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166977 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166977 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166977 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.166977 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnwg\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.166982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167264 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167493 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167493 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167493 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.167717 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.167696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.169201 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.169179 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.169317 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.169232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.174903 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.174875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.175055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.175038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnwg\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg\") pod \"e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.272907 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.272827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:40.553386 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.553354 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:15:40.554660 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:15:40.554635 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9be727_dfa7_40ca_b07e_2e157f59f5d5.slice/crio-413b32a8103aa2d5481c5e3131710bfd86afcbd10fa0684085aea33365cf0d50 WatchSource:0}: Error finding container 413b32a8103aa2d5481c5e3131710bfd86afcbd10fa0684085aea33365cf0d50: Status 404 returned error can't find the container with id 413b32a8103aa2d5481c5e3131710bfd86afcbd10fa0684085aea33365cf0d50 Sep 09 16:15:40.557422 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.557380 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:40.557490 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.557459 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:40.557540 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:40.557490 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:15:41.413272 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:41.413233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerStarted","Data":"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e"} Sep 09 16:15:41.413656 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:41.413289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerStarted","Data":"413b32a8103aa2d5481c5e3131710bfd86afcbd10fa0684085aea33365cf0d50"} Sep 09 16:15:41.433921 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:41.433870 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podStartSLOduration=2.4338566139999998 podStartE2EDuration="2.433856614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:41.432796936 +0000 UTC m=+1460.773988768" watchObservedRunningTime="2026-09-09 16:15:41.433856614 +0000 UTC m=+1460.775048442" Sep 09 16:15:42.273219 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:42.273187 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:42.278079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:42.278056 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:42.420082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:42.420055 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:42.421043 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:42.421023 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:15:53.140909 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:15:53.140871 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:16:03.555945 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.555912 2562 generic.go:356] "Generic (PLEG): container finished" podID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerID="2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e" exitCode=0 Sep 09 16:16:03.556474 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.555963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerDied","Data":"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e"} Sep 09 16:16:03.556566 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.556547 2562 scope.go:117] "RemoveContainer" containerID="2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e" Sep 09 16:16:03.557705 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.557674 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:16:03.557776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.557748 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:16:03.557822 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:03.557782 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:16:04.564889 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:04.564851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerStarted","Data":"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b"} Sep 09 16:16:05.273875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:05.273838 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:16:05.274105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:05.274082 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 09 16:16:05.274154 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:05.274131 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 09 16:16:06.274079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:06.274041 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 09 16:16:06.274492 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:06.274098 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 09 16:16:07.274209 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:07.274162 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 09 16:16:07.274618 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:07.274233 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 09 16:16:08.273076 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:08.273043 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 09 16:16:08.273245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:08.273096 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 09 16:16:09.273858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:09.273824 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 09 16:16:09.274225 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:09.273875 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 09 16:16:09.453320 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:09.453285 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:16:10.277230 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:10.277201 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:16:10.277637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:10.277458 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:16:10.278065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:10.278043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:16:21.329762 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:21.329730 2562 scope.go:117] "RemoveContainer" containerID="2d61a226095516c8fc060b014bf3f0382cfa0795cd4a0f050824cd9963b184f3" Sep 09 16:16:21.338363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:21.338343 2562 scope.go:117] "RemoveContainer" containerID="f27864883b0c271883c4a0288f8a1a448766682bffcc040ddd5b8f7df9820440" Sep 09 16:16:23.721923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:23.721888 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:16:23.725236 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:23.725210 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:16:26.124129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.124093 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:16:26.128430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.128392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131335 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131381 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131442 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj22b\" (UniqueName: \"kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131499 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.131548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.131534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.132380 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.132358 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4c4ad334\"/\"default-dockercfg-m4bmj\"" Sep 09 16:16:26.132522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.132474 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4c4ad334\"/\"e2e-lifecycle-model-4c4ad334-kserve-self-signed-certs\"" Sep 09 16:16:26.132522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.132489 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4c4ad334\"/\"kube-root-ca.crt\"" Sep 09 16:16:26.132651 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.132528 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4c4ad334\"/\"openshift-service-ca.crt\"" Sep 09 16:16:26.137481 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.137381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:16:26.231991 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.231964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232151 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232151 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj22b\" (UniqueName: \"kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232245 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232298 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232452 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232321 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232452 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232370 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.232745 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.232725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.234364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.234339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.234663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.234648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.240094 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.240053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj22b\" (UniqueName: \"kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b\") pod \"e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.439711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.439638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:26.712764 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:26.712739 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:16:26.714891 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:16:26.714866 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd550482_7491_4375_8319_de326263928b.slice/crio-4c70c9ff4b2decf8fba9790b33153110031b443727ee2fcd8fb302008e8ccf1e WatchSource:0}: Error finding container 4c70c9ff4b2decf8fba9790b33153110031b443727ee2fcd8fb302008e8ccf1e: Status 404 returned error can't find the container with id 4c70c9ff4b2decf8fba9790b33153110031b443727ee2fcd8fb302008e8ccf1e Sep 09 16:16:27.711974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:27.711944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" event={"ID":"fd550482-7491-4375-8319-de326263928b","Type":"ContainerStarted","Data":"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2"} Sep 09 16:16:27.711974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:27.711981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" event={"ID":"fd550482-7491-4375-8319-de326263928b","Type":"ContainerStarted","Data":"4c70c9ff4b2decf8fba9790b33153110031b443727ee2fcd8fb302008e8ccf1e"} Sep 09 16:16:30.040059 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:30.039999 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" podStartSLOduration=4.039981327 podStartE2EDuration="4.039981327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:16:27.731805527 +0000 UTC m=+1507.072997354" watchObservedRunningTime="2026-09-09 16:16:30.039981327 +0000 UTC m=+1509.381173154" Sep 09 16:16:30.040715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:30.040693 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:16:36.440081 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:36.440043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:36.440464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:36.440092 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:36.452354 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:36.452332 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:36.777099 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:36.777019 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:16:48.730295 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:16:48.730264 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:17:07.033585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:07.033550 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:17:13.433954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.433915 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz"] Sep 09 16:17:13.438105 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.438086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:13.441073 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.441051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:17:13.441179 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.441117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:17:13.447947 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.447921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz"] Sep 09 16:17:13.517082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.517039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz\" (UID: \"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:13.617935 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.617896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz\" (UID: \"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:13.618585 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.618565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz\" (UID: \"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:13.750233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:13.750147 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:14.024091 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:14.024065 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz"] Sep 09 16:17:14.025952 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:17:14.025920 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec2fe84_2e14_41a9_87c0_d46cd1ddb91a.slice/crio-23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3 WatchSource:0}: Error finding container 23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3: Status 404 returned error can't find the container with id 23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3 Sep 09 16:17:14.995742 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:14.995664 2562 generic.go:356] "Generic (PLEG): container finished" podID="fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" containerID="c088afb8ffdd689708f4b1060fdb1cda12e2ce9adf77da11d8f1809a8356f66e" exitCode=0 Sep 09 16:17:14.996090 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:14.995735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" event={"ID":"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a","Type":"ContainerDied","Data":"c088afb8ffdd689708f4b1060fdb1cda12e2ce9adf77da11d8f1809a8356f66e"} Sep 09 16:17:14.996090 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:14.995784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" event={"ID":"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a","Type":"ContainerStarted","Data":"23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3"} Sep 09 16:17:16.140685 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:16.140657 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:16.247663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:16.247616 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca\") pod \"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a\" (UID: \"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a\") " Sep 09 16:17:16.248015 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:16.247991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" (UID: "fec2fe84-2e14-41a9-87c0-d46cd1ddb91a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:17:16.348861 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:16.348832 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:17.008531 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:17.008496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" event={"ID":"fec2fe84-2e14-41a9-87c0-d46cd1ddb91a","Type":"ContainerDied","Data":"23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3"} Sep 09 16:17:17.008531 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:17.008530 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz" Sep 09 16:17:17.008736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:17.008546 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23f32ecbb48ac5636b42bf8745ee87b1ac163a3939e11a9d827fe19472fd98e3" Sep 09 16:17:21.365644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:21.365615 2562 scope.go:117] "RemoveContainer" containerID="536d5caa0a9f1e911e5370f908a7e421c9b25df614dbd7d4586f3e1102e1b7af" Sep 09 16:17:21.374650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:21.374628 2562 scope.go:117] "RemoveContainer" containerID="be608f0fcffcee7911169d9319078554a0fe848ec10e9c1c7f2779e65bc36085" Sep 09 16:17:23.587184 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:23.587149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:17:29.538967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:29.538925 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz"] Sep 09 16:17:29.541145 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:29.541121 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c4ad334-2fsbz"] Sep 09 16:17:31.190226 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:31.190195 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" path="/var/lib/kubelet/pods/fec2fe84-2e14-41a9-87c0-d46cd1ddb91a/volumes" Sep 09 16:17:37.694569 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:37.694532 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:17:42.232766 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:42.232729 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:17:52.714927 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:52.714868 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" podUID="fd550482-7491-4375-8319-de326263928b" containerName="main" containerID="cri-o://3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2" gracePeriod=45 Sep 09 16:17:52.962914 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:52.962891 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:17:53.036628 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036552 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.036628 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036603 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.036841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036691 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj22b\" (UniqueName: \"kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.036841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.036841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036740 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.036841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036800 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs\") pod \"fd550482-7491-4375-8319-de326263928b\" (UID: \"fd550482-7491-4375-8319-de326263928b\") " Sep 09 16:17:53.037048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036861 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:17:53.037048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:17:53.037048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.036965 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home" (OuterVolumeSpecName: "home") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:17:53.037210 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.037177 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.037255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.037206 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.037255 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.037223 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.038875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.038851 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:17:53.038875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.038858 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm" (OuterVolumeSpecName: "dshm") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:17:53.039062 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.038879 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b" (OuterVolumeSpecName: "kube-api-access-bj22b") pod "fd550482-7491-4375-8319-de326263928b" (UID: "fd550482-7491-4375-8319-de326263928b"). InnerVolumeSpecName "kube-api-access-bj22b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:17:53.138758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.138721 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj22b\" (UniqueName: \"kubernetes.io/projected/fd550482-7491-4375-8319-de326263928b-kube-api-access-bj22b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.138758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.138753 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd550482-7491-4375-8319-de326263928b-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.138758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.138764 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd550482-7491-4375-8319-de326263928b-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:17:53.237301 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.237263 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd550482-7491-4375-8319-de326263928b" containerID="3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2" exitCode=0 Sep 09 16:17:53.237454 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.237310 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" Sep 09 16:17:53.237454 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.237315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" event={"ID":"fd550482-7491-4375-8319-de326263928b","Type":"ContainerDied","Data":"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2"} Sep 09 16:17:53.237454 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.237427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5" event={"ID":"fd550482-7491-4375-8319-de326263928b","Type":"ContainerDied","Data":"4c70c9ff4b2decf8fba9790b33153110031b443727ee2fcd8fb302008e8ccf1e"} Sep 09 16:17:53.237454 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.237451 2562 scope.go:117] "RemoveContainer" containerID="3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2" Sep 09 16:17:53.246668 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.246639 2562 scope.go:117] "RemoveContainer" containerID="3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2" Sep 09 16:17:53.246961 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:17:53.246941 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2\": container with ID starting with 3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2 not found: ID does not exist" containerID="3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2" Sep 09 16:17:53.247019 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.246970 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2"} err="failed to get container status \"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2\": rpc error: code = NotFound desc = could not find container \"3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2\": container with ID starting with 3a01531a7d42415795e801c7957308f0433c7a31de1345893bda50ba715269d2 not found: ID does not exist" Sep 09 16:17:53.256943 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.256920 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:17:53.263017 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:53.262997 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4c4ad334/e2e-lifecycle-model-4c4ad334-kserve-76c8c76fff-j4qs5"] Sep 09 16:17:55.190116 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:55.190079 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd550482-7491-4375-8319-de326263928b" path="/var/lib/kubelet/pods/fd550482-7491-4375-8319-de326263928b/volumes" Sep 09 16:17:58.136212 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:17:58.136180 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:18:14.629630 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:14.629539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:18:19.183149 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:19.183116 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:18:20.892202 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:20.892166 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:18:20.892604 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:20.892419 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" containerID="cri-o://b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b" gracePeriod=30 Sep 09 16:18:21.400626 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:21.400599 2562 scope.go:117] "RemoveContainer" containerID="cc863a47990022fbd89069f0314851191d64ed5c74b0051c1b9d9791602a9467" Sep 09 16:18:21.409378 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:21.409347 2562 scope.go:117] "RemoveContainer" containerID="ab676a826fc5955e3f5ea7dd41c742107d7ddde744fa039728f554e307f2ef8f" Sep 09 16:18:25.279865 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:25.279821 2562 patch_prober.go:28] interesting pod/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:18:25.280242 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:25.279888 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:18:26.152225 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.152158 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:18:26.273976 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.273938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.273997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274067 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274144 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274208 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slnwg\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274202 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:26.274267 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274242 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:26.274757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274260 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs\") pod \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\" (UID: \"eb9be727-dfa7-40ca-b07e-2e157f59f5d5\") " Sep 09 16:18:26.274757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274463 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:26.274757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274543 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:18:26.274757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274744 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:26.275025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274872 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.275025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274896 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.275025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274912 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.275025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274926 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.275025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.274940 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.276365 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.276332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg" (OuterVolumeSpecName: "kube-api-access-slnwg") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "kube-api-access-slnwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:18:26.276494 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.276343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:18:26.276494 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.276366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:26.276494 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.276444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb9be727-dfa7-40ca-b07e-2e157f59f5d5" (UID: "eb9be727-dfa7-40ca-b07e-2e157f59f5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:18:26.376526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.376495 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.376526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.376522 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.376526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.376531 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnwg\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-kube-api-access-slnwg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.376964 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.376539 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9be727-dfa7-40ca-b07e-2e157f59f5d5-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:26.430790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.430697 2562 generic.go:356] "Generic (PLEG): container finished" podID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerID="b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b" exitCode=0 Sep 09 16:18:26.430790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.430755 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" Sep 09 16:18:26.430790 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.430780 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerDied","Data":"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b"} Sep 09 16:18:26.431056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.430834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd" event={"ID":"eb9be727-dfa7-40ca-b07e-2e157f59f5d5","Type":"ContainerDied","Data":"413b32a8103aa2d5481c5e3131710bfd86afcbd10fa0684085aea33365cf0d50"} Sep 09 16:18:26.431056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.430855 2562 scope.go:117] "RemoveContainer" containerID="b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b" Sep 09 16:18:26.440771 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.440748 2562 scope.go:117] "RemoveContainer" containerID="2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e" Sep 09 16:18:26.449096 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.449079 2562 scope.go:117] "RemoveContainer" containerID="b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b" Sep 09 16:18:26.449359 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:18:26.449342 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b\": container with ID starting with b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b not found: ID does not exist" containerID="b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b" Sep 09 16:18:26.449419 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.449370 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b"} err="failed to get container status \"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b\": rpc error: code = NotFound desc = could not find container \"b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b\": container with ID starting with b52597cb4462bab4f0bf956db68a4011d09e2ad8faeb75c886a32d1699bf704b not found: ID does not exist" Sep 09 16:18:26.449419 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.449385 2562 scope.go:117] "RemoveContainer" containerID="2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e" Sep 09 16:18:26.449646 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:18:26.449627 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e\": container with ID starting with 2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e not found: ID does not exist" containerID="2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e" Sep 09 16:18:26.449697 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.449649 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e"} err="failed to get container status \"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e\": rpc error: code = NotFound desc = could not find container \"2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e\": container with ID starting with 2f7e1d1856363fa918c23414fd331fe847219078447253e47f6f016e8bef495e not found: ID does not exist" Sep 09 16:18:26.453878 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.453856 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:18:26.457592 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:26.457571 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c4ad334-openshift-default-6856fc894d-6mdrd"] Sep 09 16:18:27.189862 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:27.189829 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" path="/var/lib/kubelet/pods/eb9be727-dfa7-40ca-b07e-2e157f59f5d5/volumes" Sep 09 16:18:29.531128 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:29.531097 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:18:34.203405 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.203348 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" podUID="590c553e-17c6-4db7-b937-d463e04bd6b6" containerName="main" containerID="cri-o://d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f" gracePeriod=45 Sep 09 16:18:34.457675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.457616 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:18:34.487339 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.487301 2562 generic.go:356] "Generic (PLEG): container finished" podID="590c553e-17c6-4db7-b937-d463e04bd6b6" containerID="d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f" exitCode=0 Sep 09 16:18:34.487507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.487341 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" Sep 09 16:18:34.487507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.487378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" event={"ID":"590c553e-17c6-4db7-b937-d463e04bd6b6","Type":"ContainerDied","Data":"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f"} Sep 09 16:18:34.487507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.487437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl" event={"ID":"590c553e-17c6-4db7-b937-d463e04bd6b6","Type":"ContainerDied","Data":"1f2cadf18f207689353b9ca1404ba1b391362d6cab596d572de7bb23362aa277"} Sep 09 16:18:34.487507 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.487462 2562 scope.go:117] "RemoveContainer" containerID="d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f" Sep 09 16:18:34.496611 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.496584 2562 scope.go:117] "RemoveContainer" containerID="d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f" Sep 09 16:18:34.496921 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:18:34.496897 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f\": container with ID starting with d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f not found: ID does not exist" containerID="d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f" Sep 09 16:18:34.496989 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.496934 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f"} err="failed to get container status \"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f\": rpc error: code = NotFound desc = could not find container \"d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f\": container with ID starting with d3b73ed5a3ef8deb8486c7a828640e11e2d719337df8b8f11d4d7c15acce0a6f not found: ID does not exist" Sep 09 16:18:34.551499 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551514 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551820 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551675 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blb6n\" (UniqueName: \"kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n\") pod \"590c553e-17c6-4db7-b937-d463e04bd6b6\" (UID: \"590c553e-17c6-4db7-b937-d463e04bd6b6\") " Sep 09 16:18:34.551820 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551753 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home" (OuterVolumeSpecName: "home") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:34.551927 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551830 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:34.551927 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.551840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:34.552093 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.552072 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.552165 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.552102 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.552165 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.552122 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.553773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.553751 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm" (OuterVolumeSpecName: "dshm") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:18:34.553891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.553871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n" (OuterVolumeSpecName: "kube-api-access-blb6n") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "kube-api-access-blb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:18:34.553969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.553872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "590c553e-17c6-4db7-b937-d463e04bd6b6" (UID: "590c553e-17c6-4db7-b937-d463e04bd6b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:18:34.652737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.652700 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590c553e-17c6-4db7-b937-d463e04bd6b6-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.652737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.652730 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590c553e-17c6-4db7-b937-d463e04bd6b6-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.652737 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.652741 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blb6n\" (UniqueName: \"kubernetes.io/projected/590c553e-17c6-4db7-b937-d463e04bd6b6-kube-api-access-blb6n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:18:34.809303 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.809274 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:18:34.812289 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:34.812262 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/auth-test-model-131689-kserve-6ff7577c9-w5htl"] Sep 09 16:18:35.191807 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:35.191771 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590c553e-17c6-4db7-b937-d463e04bd6b6" path="/var/lib/kubelet/pods/590c553e-17c6-4db7-b937-d463e04bd6b6/volumes" Sep 09 16:18:38.829969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:38.829931 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:18:41.337348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337314 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337761 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" containerName="revoke-keys" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337776 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" containerName="revoke-keys" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337793 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd550482-7491-4375-8319-de326263928b" containerName="main" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337799 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd550482-7491-4375-8319-de326263928b" containerName="main" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337809 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337814 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337827 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337832 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337839 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590c553e-17c6-4db7-b937-d463e04bd6b6" containerName="main" Sep 09 16:18:41.337848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337844 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="590c553e-17c6-4db7-b937-d463e04bd6b6" containerName="main" Sep 09 16:18:41.338355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337901 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.338355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337909 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9be727-dfa7-40ca-b07e-2e157f59f5d5" containerName="istio-proxy" Sep 09 16:18:41.338355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337918 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="590c553e-17c6-4db7-b937-d463e04bd6b6" containerName="main" Sep 09 16:18:41.338355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337923 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd550482-7491-4375-8319-de326263928b" containerName="main" Sep 09 16:18:41.338355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.337930 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec2fe84-2e14-41a9-87c0-d46cd1ddb91a" containerName="revoke-keys" Sep 09 16:18:41.342213 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.342192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.345133 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.345111 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"kube-root-ca.crt\"" Sep 09 16:18:41.346175 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.346159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"openshift-service-ca.crt\"" Sep 09 16:18:41.346527 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.346506 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"sub-test-model-131689-kserve-self-signed-certs\"" Sep 09 16:18:41.346652 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.346506 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-131689\"/\"default-dockercfg-m6knh\"" Sep 09 16:18:41.350689 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.350671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:18:41.521669 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phtl\" (UniqueName: \"kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.521840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.521840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.521840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.521964 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.521964 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.521923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622490 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622680 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622680 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622680 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622855 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phtl\" (UniqueName: \"kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.622855 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.622748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.623103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.623073 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.623333 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.623301 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.623460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.623339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.625078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.625057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.625304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.625280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.631436 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.631400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phtl\" (UniqueName: \"kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl\") pod \"sub-test-model-131689-kserve-7887b96575-9n5r5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.653564 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.653176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:41.927573 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:41.927533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:18:41.928495 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:18:41.928462 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dc12ec8_62b3_4558_b8ba_727315236af5.slice/crio-668008fb17c927435d85d0043ea7ecc73b2300ccb2a0d65b0899e29b8247ce0b WatchSource:0}: Error finding container 668008fb17c927435d85d0043ea7ecc73b2300ccb2a0d65b0899e29b8247ce0b: Status 404 returned error can't find the container with id 668008fb17c927435d85d0043ea7ecc73b2300ccb2a0d65b0899e29b8247ce0b Sep 09 16:18:42.533831 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:42.533798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" event={"ID":"3dc12ec8-62b3-4558-b8ba-727315236af5","Type":"ContainerStarted","Data":"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0"} Sep 09 16:18:42.534213 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:42.533841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" event={"ID":"3dc12ec8-62b3-4558-b8ba-727315236af5","Type":"ContainerStarted","Data":"668008fb17c927435d85d0043ea7ecc73b2300ccb2a0d65b0899e29b8247ce0b"} Sep 09 16:18:42.550809 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:42.550758 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" podStartSLOduration=1.550734983 podStartE2EDuration="1.550734983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:42.549886902 +0000 UTC m=+1641.891078731" watchObservedRunningTime="2026-09-09 16:18:42.550734983 +0000 UTC m=+1641.891926813" Sep 09 16:18:51.654202 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:51.654172 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:51.654624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:51.654218 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:51.666645 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:51.666619 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:18:52.028624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:52.028538 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:18:52.600647 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:18:52.600611 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:19:15.441393 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.441361 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:19:15.445541 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.445520 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.449152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.449127 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-94cf14-openshift-default-dockercfg-4cpkn\"" Sep 09 16:19:15.458503 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.458482 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:19:15.556208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxgf\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556765 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556765 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.556765 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.556530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxgf\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657523 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657523 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657655 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657655 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657768 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657979 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.657979 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.658143 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.657984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.658143 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.658070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.658363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.658345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.659858 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.659838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.660040 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.660022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.666999 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.666978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.667205 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.667187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxgf\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf\") pod \"e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:15.759881 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:15.759808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:16.041505 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.041484 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:19:16.044129 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:19:16.044100 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce8ff2c_41ff_44ae_990b_c7c5a20a6630.slice/crio-7fa06a0c921168ffb7d539f83caeefcfc5bfb4f5191599f091acda90150453f2 WatchSource:0}: Error finding container 7fa06a0c921168ffb7d539f83caeefcfc5bfb4f5191599f091acda90150453f2: Status 404 returned error can't find the container with id 7fa06a0c921168ffb7d539f83caeefcfc5bfb4f5191599f091acda90150453f2 Sep 09 16:19:16.046677 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.046637 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:16.046771 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.046750 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:16.046813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.046800 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:16.730584 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.730550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerStarted","Data":"d74271ed36a2028e4a618b760e42c0134ce42b7419112c121d74a52a45d39cbc"} Sep 09 16:19:16.731012 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.730603 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerStarted","Data":"7fa06a0c921168ffb7d539f83caeefcfc5bfb4f5191599f091acda90150453f2"} Sep 09 16:19:16.759938 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.759890 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podStartSLOduration=1.759874138 podStartE2EDuration="1.759874138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:16.758499335 +0000 UTC m=+1676.099691164" watchObservedRunningTime="2026-09-09 16:19:16.759874138 +0000 UTC m=+1676.101065965" Sep 09 16:19:16.760486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:16.760459 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:17.765131 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:17.765095 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:18.742576 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:18.742540 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:18.743707 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:18.743681 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:26.548173 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.548137 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:19:26.552026 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.552004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.554775 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.554752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"default-dockercfg-898cn\"" Sep 09 16:19:26.555894 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.555875 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"kube-root-ca.crt\"" Sep 09 16:19:26.556010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.555911 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"sub-test-model-13d59f-kserve-self-signed-certs\"" Sep 09 16:19:26.556010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.555881 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"openshift-service-ca.crt\"" Sep 09 16:19:26.562336 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.562313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:19:26.575129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.575216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.575257 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.575296 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.575296 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.575375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.575306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tpc\" (UniqueName: \"kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.676840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.676802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.676846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.676865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.676888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tpc\" (UniqueName: \"kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.676938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.677089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677310 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.677287 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677354 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.677307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.677390 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.677351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.679216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.679193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.679358 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.679341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.686391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.686368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tpc\" (UniqueName: \"kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc\") pod \"sub-test-model-13d59f-kserve-6f4d4769f-w82vl\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:26.862815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:26.862779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:27.136577 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:27.136542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:19:27.137304 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:19:27.137280 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57056a16_251c_4e42_8cbb_8cc182da9ef6.slice/crio-28d0d1cfd42795257cd8c4404d4a61a889c880657bc8a7faeac6f611993c50f3 WatchSource:0}: Error finding container 28d0d1cfd42795257cd8c4404d4a61a889c880657bc8a7faeac6f611993c50f3: Status 404 returned error can't find the container with id 28d0d1cfd42795257cd8c4404d4a61a889c880657bc8a7faeac6f611993c50f3 Sep 09 16:19:27.803519 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:27.803482 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" event={"ID":"57056a16-251c-4e42-8cbb-8cc182da9ef6","Type":"ContainerStarted","Data":"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e"} Sep 09 16:19:27.803519 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:27.803522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" event={"ID":"57056a16-251c-4e42-8cbb-8cc182da9ef6","Type":"ContainerStarted","Data":"28d0d1cfd42795257cd8c4404d4a61a889c880657bc8a7faeac6f611993c50f3"} Sep 09 16:19:27.823537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:27.823486 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" podStartSLOduration=1.823470004 podStartE2EDuration="1.823470004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:27.821863601 +0000 UTC m=+1687.163055429" watchObservedRunningTime="2026-09-09 16:19:27.823470004 +0000 UTC m=+1687.164661831" Sep 09 16:19:29.942623 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:29.942588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:19:36.863241 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:36.863210 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:36.863711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:36.863267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:36.875819 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:36.875794 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:37.874210 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:37.874177 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:19:40.880795 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.880760 2562 generic.go:356] "Generic (PLEG): container finished" podID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerID="d74271ed36a2028e4a618b760e42c0134ce42b7419112c121d74a52a45d39cbc" exitCode=0 Sep 09 16:19:40.881220 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.880832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerDied","Data":"d74271ed36a2028e4a618b760e42c0134ce42b7419112c121d74a52a45d39cbc"} Sep 09 16:19:40.881281 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.881237 2562 scope.go:117] "RemoveContainer" containerID="d74271ed36a2028e4a618b760e42c0134ce42b7419112c121d74a52a45d39cbc" Sep 09 16:19:40.882358 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.882325 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:40.882482 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.882429 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:40.882557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:40.882482 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:41.893759 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:41.893712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerStarted","Data":"5a6f6dc201715b85d296b4bd52661f08afb9aeb214fa390744607206dd753d41"} Sep 09 16:19:42.760711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:42.760669 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:42.761028 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:42.761002 2562 patch_prober.go:28] interesting pod/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 09 16:19:42.761078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:42.761060 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 09 16:19:43.760431 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:43.760332 2562 patch_prober.go:28] interesting pod/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 09 16:19:43.760431 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:43.760393 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 09 16:19:44.413889 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.413858 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:19:44.416526 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.416507 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.419393 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.419369 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-07cc17-openshift-default-dockercfg-2w4vh\"" Sep 09 16:19:44.429733 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.429708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:19:44.562427 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562617 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x75q\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.562921 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.562784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664635 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x75q\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664857 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664857 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664857 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.664857 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.664806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.665072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.665048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.665181 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.665159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.665302 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.665208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.667024 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.667007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.667085 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.667042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.672757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.672724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.672927 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.672911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x75q\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q\") pod \"e2e-api-b-07cc17-openshift-default-8496c8c896-229lr\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.730658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.730624 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:44.761121 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.761090 2562 patch_prober.go:28] interesting pod/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 09 16:19:44.761463 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:44.761158 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 09 16:19:45.023569 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.023539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:19:45.025275 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:19:45.025250 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f2d08_2643_4607_8ee6_c34e983d4d8e.slice/crio-fe5555a8ffdf41e38096b81bad9124dd600d99ab90e7c59672a2a2c4194c1415 WatchSource:0}: Error finding container fe5555a8ffdf41e38096b81bad9124dd600d99ab90e7c59672a2a2c4194c1415: Status 404 returned error can't find the container with id fe5555a8ffdf41e38096b81bad9124dd600d99ab90e7c59672a2a2c4194c1415 Sep 09 16:19:45.028086 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.028056 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:45.028160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.028120 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:45.028160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.028151 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:19:45.760346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.760307 2562 patch_prober.go:28] interesting pod/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 09 16:19:45.760544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.760368 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 09 16:19:45.921842 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.921806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerStarted","Data":"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329"} Sep 09 16:19:45.922237 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.921874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerStarted","Data":"fe5555a8ffdf41e38096b81bad9124dd600d99ab90e7c59672a2a2c4194c1415"} Sep 09 16:19:45.955603 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:45.955554 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podStartSLOduration=1.9555391100000001 podStartE2EDuration="1.95553911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:45.953740424 +0000 UTC m=+1705.294932253" watchObservedRunningTime="2026-09-09 16:19:45.95553911 +0000 UTC m=+1705.296730938" Sep 09 16:19:46.731593 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.731554 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:46.737235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.737209 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:46.760569 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.760535 2562 patch_prober.go:28] interesting pod/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 09 16:19:46.760709 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.760593 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 09 16:19:46.927568 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.927537 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:46.928551 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:46.928527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:19:47.764952 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:47.764924 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:47.765323 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:47.765307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:47.766025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:47.766004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:19:51.142838 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:19:51.142802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:20:10.382434 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:10.381278 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:20:11.517320 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:11.517288 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:20:13.086470 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.086433 2562 generic.go:356] "Generic (PLEG): container finished" podID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerID="f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329" exitCode=0 Sep 09 16:20:13.086878 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.086483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerDied","Data":"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329"} Sep 09 16:20:13.086951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.086936 2562 scope.go:117] "RemoveContainer" containerID="f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329" Sep 09 16:20:13.088160 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.088126 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:20:13.088259 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.088197 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:20:13.088259 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:13.088229 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:20:14.096177 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:14.096135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerStarted","Data":"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3"} Sep 09 16:20:14.131895 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:14.131841 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:20:14.731360 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:14.731327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:20:14.732715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:14.732684 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 09 16:20:14.732854 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:14.732737 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 09 16:20:15.687470 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.687431 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd"] Sep 09 16:20:15.691011 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.690980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:15.696806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.696773 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:20:15.697037 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.697015 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:20:15.699741 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.699714 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd"] Sep 09 16:20:15.731488 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.731463 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 09 16:20:15.731608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.731522 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 09 16:20:15.820456 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.820418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd\" (UID: \"7c99d42d-2720-4937-a93b-8dd847f9c7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:15.921605 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.921571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd\" (UID: \"7c99d42d-2720-4937-a93b-8dd847f9c7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:15.922216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:15.922194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd\" (UID: \"7c99d42d-2720-4937-a93b-8dd847f9c7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:16.007421 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:16.007305 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:16.288492 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:16.286131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd"] Sep 09 16:20:16.731346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:16.731315 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 09 16:20:16.731743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:16.731367 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 09 16:20:16.928122 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:16.928092 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:20:17.121781 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:17.121743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" event={"ID":"7c99d42d-2720-4937-a93b-8dd847f9c7c4","Type":"ContainerStarted","Data":"06c933e9be532585a5e83865dee2e83dfc83b55fb8df28b07f774b5c9cf3c134"} Sep 09 16:20:17.122000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:17.121807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" event={"ID":"7c99d42d-2720-4937-a93b-8dd847f9c7c4","Type":"ContainerStarted","Data":"2db8cdfeae11f8d0ff13762912f333fcc273490b5f2835fa5a9c10c5cfda60f3"} Sep 09 16:20:17.141317 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:17.141271 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" podStartSLOduration=2.141253115 podStartE2EDuration="2.141253115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:17.139488902 +0000 UTC m=+1736.480680730" watchObservedRunningTime="2026-09-09 16:20:17.141253115 +0000 UTC m=+1736.482444944" Sep 09 16:20:17.323951 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:20:17.323916 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c99d42d_2720_4937_a93b_8dd847f9c7c4.slice/crio-06c933e9be532585a5e83865dee2e83dfc83b55fb8df28b07f774b5c9cf3c134.scope\": RecentStats: unable to find data in memory cache]" Sep 09 16:20:17.731788 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:17.731750 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 09 16:20:17.732171 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:17.731809 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 09 16:20:18.128699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:18.128665 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c99d42d-2720-4937-a93b-8dd847f9c7c4" containerID="06c933e9be532585a5e83865dee2e83dfc83b55fb8df28b07f774b5c9cf3c134" exitCode=0 Sep 09 16:20:18.128881 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:18.128741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" event={"ID":"7c99d42d-2720-4937-a93b-8dd847f9c7c4","Type":"ContainerDied","Data":"06c933e9be532585a5e83865dee2e83dfc83b55fb8df28b07f774b5c9cf3c134"} Sep 09 16:20:18.731998 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:18.731963 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 09 16:20:18.732429 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:18.732020 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 09 16:20:19.319707 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.319675 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:19.460649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.460544 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca\") pod \"7c99d42d-2720-4937-a93b-8dd847f9c7c4\" (UID: \"7c99d42d-2720-4937-a93b-8dd847f9c7c4\") " Sep 09 16:20:19.460894 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.460869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c99d42d-2720-4937-a93b-8dd847f9c7c4" (UID: "7c99d42d-2720-4937-a93b-8dd847f9c7c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:20:19.460993 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.460981 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c99d42d-2720-4937-a93b-8dd847f9c7c4-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:19.745517 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.745400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:20:19.746561 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:19.746539 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:20:20.142224 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:20.142195 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" Sep 09 16:20:20.142397 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:20.142234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd" event={"ID":"7c99d42d-2720-4937-a93b-8dd847f9c7c4","Type":"ContainerDied","Data":"2db8cdfeae11f8d0ff13762912f333fcc273490b5f2835fa5a9c10c5cfda60f3"} Sep 09 16:20:20.142397 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:20.142310 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db8cdfeae11f8d0ff13762912f333fcc273490b5f2835fa5a9c10c5cfda60f3" Sep 09 16:20:21.457930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:21.457886 2562 scope.go:117] "RemoveContainer" containerID="a89db23c42dc0e3d7e6853500f9ad935ed7eb8b7bea3933c1ec9801e7f0cfe3b" Sep 09 16:20:21.467109 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:21.467084 2562 scope.go:117] "RemoveContainer" containerID="9368336cb92ee557c920f475f4c59f8a4d1000c837dc197b6c4a3ea4a159d70e" Sep 09 16:20:25.402868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.402814 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" podUID="3dc12ec8-62b3-4558-b8ba-727315236af5" containerName="main" containerID="cri-o://5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0" gracePeriod=45 Sep 09 16:20:25.661984 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.661928 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:20:25.721286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721253 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721286 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721290 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721306 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721344 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5phtl\" (UniqueName: \"kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721543 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721504 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home\") pod \"3dc12ec8-62b3-4558-b8ba-727315236af5\" (UID: \"3dc12ec8-62b3-4558-b8ba-727315236af5\") " Sep 09 16:20:25.721773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721578 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:25.721773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:25.721883 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home" (OuterVolumeSpecName: "home") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:25.721954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721937 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:25.722012 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721955 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:25.722012 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.721963 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:25.723570 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.723540 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:20:25.723570 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.723547 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm" (OuterVolumeSpecName: "dshm") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:25.723747 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.723565 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl" (OuterVolumeSpecName: "kube-api-access-5phtl") pod "3dc12ec8-62b3-4558-b8ba-727315236af5" (UID: "3dc12ec8-62b3-4558-b8ba-727315236af5"). InnerVolumeSpecName "kube-api-access-5phtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:20:25.822711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.822672 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5phtl\" (UniqueName: \"kubernetes.io/projected/3dc12ec8-62b3-4558-b8ba-727315236af5-kube-api-access-5phtl\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:25.822711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.822704 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc12ec8-62b3-4558-b8ba-727315236af5-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:25.822711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:25.822714 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc12ec8-62b3-4558-b8ba-727315236af5-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.179071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.179038 2562 generic.go:356] "Generic (PLEG): container finished" podID="3dc12ec8-62b3-4558-b8ba-727315236af5" containerID="5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0" exitCode=0 Sep 09 16:20:26.179071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.179074 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" Sep 09 16:20:26.179306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.179075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" event={"ID":"3dc12ec8-62b3-4558-b8ba-727315236af5","Type":"ContainerDied","Data":"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0"} Sep 09 16:20:26.179306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.179202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5" event={"ID":"3dc12ec8-62b3-4558-b8ba-727315236af5","Type":"ContainerDied","Data":"668008fb17c927435d85d0043ea7ecc73b2300ccb2a0d65b0899e29b8247ce0b"} Sep 09 16:20:26.179306 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.179226 2562 scope.go:117] "RemoveContainer" containerID="5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0" Sep 09 16:20:26.188861 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.188840 2562 scope.go:117] "RemoveContainer" containerID="5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0" Sep 09 16:20:26.189153 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:20:26.189135 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0\": container with ID starting with 5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0 not found: ID does not exist" containerID="5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0" Sep 09 16:20:26.189207 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.189162 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0"} err="failed to get container status \"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0\": rpc error: code = NotFound desc = could not find container \"5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0\": container with ID starting with 5f723ab3b5642791c68df3f5340eb478076e8f9cf3be7e98f8dd35ef03df9cb0 not found: ID does not exist" Sep 09 16:20:26.203141 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.203115 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:20:26.205362 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.205339 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-131689/sub-test-model-131689-kserve-7887b96575-9n5r5"] Sep 09 16:20:26.537839 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.537740 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" podUID="57056a16-251c-4e42-8cbb-8cc182da9ef6" containerName="main" containerID="cri-o://ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e" gracePeriod=45 Sep 09 16:20:26.792494 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.792434 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:20:26.832195 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832162 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832245 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832333 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832355 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832377 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6tpc\" (UniqueName: \"kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc\") pod \"57056a16-251c-4e42-8cbb-8cc182da9ef6\" (UID: \"57056a16-251c-4e42-8cbb-8cc182da9ef6\") " Sep 09 16:20:26.832712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832516 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache" (OuterVolumeSpecName: "model-cache") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:26.832712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:26.832835 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832732 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home" (OuterVolumeSpecName: "home") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:26.832875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832837 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.832875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832857 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.832875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.832871 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.834683 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.834659 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc" (OuterVolumeSpecName: "kube-api-access-c6tpc") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "kube-api-access-c6tpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:20:26.834954 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.834927 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm" (OuterVolumeSpecName: "dshm") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:26.835025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.834932 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57056a16-251c-4e42-8cbb-8cc182da9ef6" (UID: "57056a16-251c-4e42-8cbb-8cc182da9ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:20:26.933884 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.933844 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57056a16-251c-4e42-8cbb-8cc182da9ef6-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.933884 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.933877 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57056a16-251c-4e42-8cbb-8cc182da9ef6-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:26.933884 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:26.933887 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tpc\" (UniqueName: \"kubernetes.io/projected/57056a16-251c-4e42-8cbb-8cc182da9ef6-kube-api-access-c6tpc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:27.186328 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.186294 2562 generic.go:356] "Generic (PLEG): container finished" podID="57056a16-251c-4e42-8cbb-8cc182da9ef6" containerID="ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e" exitCode=0 Sep 09 16:20:27.186547 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.186423 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" Sep 09 16:20:27.190825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.190794 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc12ec8-62b3-4558-b8ba-727315236af5" path="/var/lib/kubelet/pods/3dc12ec8-62b3-4558-b8ba-727315236af5/volumes" Sep 09 16:20:27.191236 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.191219 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" event={"ID":"57056a16-251c-4e42-8cbb-8cc182da9ef6","Type":"ContainerDied","Data":"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e"} Sep 09 16:20:27.191311 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.191248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl" event={"ID":"57056a16-251c-4e42-8cbb-8cc182da9ef6","Type":"ContainerDied","Data":"28d0d1cfd42795257cd8c4404d4a61a889c880657bc8a7faeac6f611993c50f3"} Sep 09 16:20:27.191311 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.191267 2562 scope.go:117] "RemoveContainer" containerID="ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e" Sep 09 16:20:27.201629 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.201611 2562 scope.go:117] "RemoveContainer" containerID="ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e" Sep 09 16:20:27.201920 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:20:27.201901 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e\": container with ID starting with ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e not found: ID does not exist" containerID="ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e" Sep 09 16:20:27.201969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.201928 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e"} err="failed to get container status \"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e\": rpc error: code = NotFound desc = could not find container \"ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e\": container with ID starting with ea534fd67ffd5cad4943a11bb97be5cfa0abfd42fd59647af27c68c22786342e not found: ID does not exist" Sep 09 16:20:27.210696 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.210670 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:20:27.212930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:27.212912 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/sub-test-model-13d59f-kserve-6f4d4769f-w82vl"] Sep 09 16:20:29.189890 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:29.189855 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57056a16-251c-4e42-8cbb-8cc182da9ef6" path="/var/lib/kubelet/pods/57056a16-251c-4e42-8cbb-8cc182da9ef6/volumes" Sep 09 16:20:31.526636 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:31.526595 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd"] Sep 09 16:20:31.528801 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:31.528779 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94cf14-xbzfd"] Sep 09 16:20:33.190011 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:33.189980 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c99d42d-2720-4937-a93b-8dd847f9c7c4" path="/var/lib/kubelet/pods/7c99d42d-2720-4937-a93b-8dd847f9c7c4/volumes" Sep 09 16:20:33.526302 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:33.526217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:20:38.065760 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.065725 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:20:38.066234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.065996 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" containerID="cri-o://5a6f6dc201715b85d296b4bd52661f08afb9aeb214fa390744607206dd753d41" gracePeriod=30 Sep 09 16:20:38.406380 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406347 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw"] Sep 09 16:20:38.406827 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406813 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc12ec8-62b3-4558-b8ba-727315236af5" containerName="main" Sep 09 16:20:38.406872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406828 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc12ec8-62b3-4558-b8ba-727315236af5" containerName="main" Sep 09 16:20:38.406872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406853 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57056a16-251c-4e42-8cbb-8cc182da9ef6" containerName="main" Sep 09 16:20:38.406872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406858 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57056a16-251c-4e42-8cbb-8cc182da9ef6" containerName="main" Sep 09 16:20:38.406974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406879 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c99d42d-2720-4937-a93b-8dd847f9c7c4" containerName="revoke-keys" Sep 09 16:20:38.406974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406884 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c99d42d-2720-4937-a93b-8dd847f9c7c4" containerName="revoke-keys" Sep 09 16:20:38.406974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406940 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57056a16-251c-4e42-8cbb-8cc182da9ef6" containerName="main" Sep 09 16:20:38.406974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406951 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc12ec8-62b3-4558-b8ba-727315236af5" containerName="main" Sep 09 16:20:38.406974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.406961 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c99d42d-2720-4937-a93b-8dd847f9c7c4" containerName="revoke-keys" Sep 09 16:20:38.411806 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.411788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:38.414654 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.414625 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:20:38.414787 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.414674 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:20:38.416878 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.416852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw"] Sep 09 16:20:38.452089 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.452057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07cc17-99hcw\" (UID: \"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:38.553773 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.553734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07cc17-99hcw\" (UID: \"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:38.554446 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.554391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-07cc17-99hcw\" (UID: \"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:38.725740 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.725644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:38.998713 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:38.998662 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw"] Sep 09 16:20:39.001220 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:20:39.001180 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cf4a19_0cfd_4a05_baeb_b22aa7f3afd4.slice/crio-05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d WatchSource:0}: Error finding container 05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d: Status 404 returned error can't find the container with id 05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d Sep 09 16:20:39.261704 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:39.261621 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" event={"ID":"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4","Type":"ContainerStarted","Data":"aee8ab186996348f66c2ff70dc2599a4b7ecfede0d021292f0d5a5bacd925981"} Sep 09 16:20:39.261704 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:39.261674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" event={"ID":"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4","Type":"ContainerStarted","Data":"05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d"} Sep 09 16:20:39.276580 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:39.276527 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" podStartSLOduration=1.27650968 podStartE2EDuration="1.27650968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:39.275084869 +0000 UTC m=+1758.616276698" watchObservedRunningTime="2026-09-09 16:20:39.27650968 +0000 UTC m=+1758.617701509" Sep 09 16:20:40.269080 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:40.269040 2562 generic.go:356] "Generic (PLEG): container finished" podID="35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" containerID="aee8ab186996348f66c2ff70dc2599a4b7ecfede0d021292f0d5a5bacd925981" exitCode=0 Sep 09 16:20:40.269522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:40.269083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" event={"ID":"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4","Type":"ContainerDied","Data":"aee8ab186996348f66c2ff70dc2599a4b7ecfede0d021292f0d5a5bacd925981"} Sep 09 16:20:41.419917 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:41.419895 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:41.482210 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:41.482176 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca\") pod \"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4\" (UID: \"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4\") " Sep 09 16:20:41.482559 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:41.482536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" (UID: "35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:20:41.583945 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:41.583856 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:42.288840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:42.288807 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" Sep 09 16:20:42.289016 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:42.288843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw" event={"ID":"35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4","Type":"ContainerDied","Data":"05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d"} Sep 09 16:20:42.289016 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:42.288927 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f191d9b0bbc2d9ccb37dc785cd3051528a8f1c34d1bd761e52c4e27754865d" Sep 09 16:20:43.297854 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.297820 2562 generic.go:356] "Generic (PLEG): container finished" podID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerID="5a6f6dc201715b85d296b4bd52661f08afb9aeb214fa390744607206dd753d41" exitCode=0 Sep 09 16:20:43.298304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.297880 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerDied","Data":"5a6f6dc201715b85d296b4bd52661f08afb9aeb214fa390744607206dd753d41"} Sep 09 16:20:43.298304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.297932 2562 scope.go:117] "RemoveContainer" containerID="d74271ed36a2028e4a618b760e42c0134ce42b7419112c121d74a52a45d39cbc" Sep 09 16:20:43.336210 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.336143 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:20:43.402144 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402090 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402176 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrxgf\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402220 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402259 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402351 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402394 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402505 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402571 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token\") pod \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\" (UID: \"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630\") " Sep 09 16:20:43.402876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402646 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:43.402876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402681 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:43.402876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402679 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:43.403110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402958 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:20:43.403110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402970 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.403110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.402989 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.403110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.403002 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.403110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.403017 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.404677 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.404653 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf" (OuterVolumeSpecName: "kube-api-access-mrxgf") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "kube-api-access-mrxgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:20:43.404800 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.404681 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:20:43.404951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.404929 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:20:43.405044 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.404947 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" (UID: "8ce8ff2c-41ff-44ae-990b-c7c5a20a6630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:20:43.504003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.503964 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.504003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.503998 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.504003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.504008 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxgf\" (UniqueName: \"kubernetes.io/projected/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-kube-api-access-mrxgf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.504242 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.504017 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:43.504242 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:43.504025 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:20:44.305072 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:44.305037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" event={"ID":"8ce8ff2c-41ff-44ae-990b-c7c5a20a6630","Type":"ContainerDied","Data":"7fa06a0c921168ffb7d539f83caeefcfc5bfb4f5191599f091acda90150453f2"} Sep 09 16:20:44.305522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:44.305095 2562 scope.go:117] "RemoveContainer" containerID="5a6f6dc201715b85d296b4bd52661f08afb9aeb214fa390744607206dd753d41" Sep 09 16:20:44.305522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:44.305044 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6" Sep 09 16:20:44.327344 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:44.327316 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:20:44.333516 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:44.333495 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-94cf14-openshift-default-d579fcdc8-9h4z6"] Sep 09 16:20:45.191346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:45.191310 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" path="/var/lib/kubelet/pods/8ce8ff2c-41ff-44ae-990b-c7c5a20a6630/volumes" Sep 09 16:20:45.627979 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:45.627942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:20:54.169284 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:54.169248 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw"] Sep 09 16:20:54.175621 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:54.175595 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-07cc17-99hcw"] Sep 09 16:20:55.190544 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:55.190508 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" path="/var/lib/kubelet/pods/35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4/volumes" Sep 09 16:20:56.539799 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.539762 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:20:56.540460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540439 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" containerName="revoke-keys" Sep 09 16:20:56.540460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540460 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" containerName="revoke-keys" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540489 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540495 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540509 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540515 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540587 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:20:56.540627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.540594 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cf4a19-0cfd-4a05-baeb-b22aa7f3afd4" containerName="revoke-keys" Sep 09 16:20:56.543875 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.543858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.547512 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.547488 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"openshift-service-ca.crt\"" Sep 09 16:20:56.547512 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.547503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"kube-root-ca.crt\"" Sep 09 16:20:56.547669 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.547503 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"default-dockercfg-898cn\"" Sep 09 16:20:56.547669 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.547502 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-13d59f\"/\"rate-test-model-13d59f-kserve-self-signed-certs\"" Sep 09 16:20:56.553006 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.552975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:20:56.636618 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.636799 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.636799 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.636882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.636882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gq2\" (UniqueName: \"kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.636949 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.636893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.737827 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gq2\" (UniqueName: \"kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737955 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738001 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.737997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738262 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.738193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.738311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.738392 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.738367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.740093 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.740068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.740337 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.740318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.746184 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.746165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gq2\" (UniqueName: \"kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2\") pod \"rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:56.854701 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:56.854668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:20:57.127476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:57.127447 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:20:57.128947 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:20:57.128910 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ec51227_909f_4bc0_bc9b_7d26781000ad.slice/crio-e45a24eddba502169c51c55b50d2e5fc571d866157f470d1e8e4c31bc5b9fbf6 WatchSource:0}: Error finding container e45a24eddba502169c51c55b50d2e5fc571d866157f470d1e8e4c31bc5b9fbf6: Status 404 returned error can't find the container with id e45a24eddba502169c51c55b50d2e5fc571d866157f470d1e8e4c31bc5b9fbf6 Sep 09 16:20:57.390672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:57.390588 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" event={"ID":"1ec51227-909f-4bc0-bc9b-7d26781000ad","Type":"ContainerStarted","Data":"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316"} Sep 09 16:20:57.390672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:57.390627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" event={"ID":"1ec51227-909f-4bc0-bc9b-7d26781000ad","Type":"ContainerStarted","Data":"e45a24eddba502169c51c55b50d2e5fc571d866157f470d1e8e4c31bc5b9fbf6"} Sep 09 16:20:57.406674 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:20:57.406626 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" podStartSLOduration=1.40661268 podStartE2EDuration="1.40661268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:57.405625131 +0000 UTC m=+1776.746816959" watchObservedRunningTime="2026-09-09 16:20:57.40661268 +0000 UTC m=+1776.747804557" Sep 09 16:21:00.343988 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.343950 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:21:00.344402 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.344214 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" containerID="cri-o://e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3" gracePeriod=30 Sep 09 16:21:00.719949 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.719870 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln"] Sep 09 16:21:00.720486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.720471 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce8ff2c-41ff-44ae-990b-c7c5a20a6630" containerName="istio-proxy" Sep 09 16:21:00.723694 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.723679 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:00.726383 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.726357 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:21:00.726539 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.726442 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:21:00.730094 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.730073 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln"] Sep 09 16:21:00.786449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.786385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln\" (UID: \"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:00.841974 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.841936 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:21:00.888093 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.888056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln\" (UID: \"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:00.888782 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:00.888756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln\" (UID: \"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:01.037856 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:01.037756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:01.311473 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:01.311442 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln"] Sep 09 16:21:01.313501 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:21:01.313471 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaae4d1b_d0b7_46e9_9ca8_e42486d9afe2.slice/crio-0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf WatchSource:0}: Error finding container 0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf: Status 404 returned error can't find the container with id 0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf Sep 09 16:21:01.414850 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:01.414806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" event={"ID":"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2","Type":"ContainerStarted","Data":"0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf"} Sep 09 16:21:02.423294 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:02.423261 2562 generic.go:356] "Generic (PLEG): container finished" podID="eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" containerID="3eabab11d4371cf3c9db80949272d7c3b26e84e5af2838ae0c0129bf6ff7303b" exitCode=0 Sep 09 16:21:02.423670 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:02.423340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" event={"ID":"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2","Type":"ContainerDied","Data":"3eabab11d4371cf3c9db80949272d7c3b26e84e5af2838ae0c0129bf6ff7303b"} Sep 09 16:21:03.567356 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:03.567327 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:03.618142 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:03.618103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca\") pod \"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2\" (UID: \"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2\") " Sep 09 16:21:03.618500 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:03.618478 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" (UID: "eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:21:03.719460 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:03.719333 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:04.436778 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:04.436743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" event={"ID":"eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2","Type":"ContainerDied","Data":"0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf"} Sep 09 16:21:04.436778 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:04.436773 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln" Sep 09 16:21:04.436999 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:04.436800 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fb72315cc1d8051c3fda656bc9ac54b7f2512e899f9331e2e8967ce0204cdcf" Sep 09 16:21:04.748473 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:04.748379 2562 patch_prober.go:28] interesting pod/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 16:21:04.748838 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:04.748470 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 16:21:05.596111 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.596084 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:21:05.638622 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638622 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638615 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638654 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638701 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638735 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638766 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x75q\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.638823 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.639079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638890 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.639079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638930 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:21:05.639079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.638964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket\") pod \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\" (UID: \"c01f2d08-2643-4607-8ee6-c34e983d4d8e\") " Sep 09 16:21:05.639239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639128 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:05.639239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639203 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:05.639239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639223 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:05.639395 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639368 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:05.639736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639714 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.639853 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639738 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.639853 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639753 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.639853 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639768 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.639853 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.639782 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.641222 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.641192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:05.641332 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.641277 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:21:05.641461 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.641404 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:21:05.641734 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.641717 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q" (OuterVolumeSpecName: "kube-api-access-5x75q") pod "c01f2d08-2643-4607-8ee6-c34e983d4d8e" (UID: "c01f2d08-2643-4607-8ee6-c34e983d4d8e"). InnerVolumeSpecName "kube-api-access-5x75q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:21:05.740627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.740588 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x75q\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-kube-api-access-5x75q\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.740627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.740623 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.740627 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.740634 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:05.740859 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:05.740642 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01f2d08-2643-4607-8ee6-c34e983d4d8e-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:06.450159 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.450115 2562 generic.go:356] "Generic (PLEG): container finished" podID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerID="e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3" exitCode=0 Sep 09 16:21:06.450650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.450190 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" Sep 09 16:21:06.450650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.450204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerDied","Data":"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3"} Sep 09 16:21:06.450650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.450289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr" event={"ID":"c01f2d08-2643-4607-8ee6-c34e983d4d8e","Type":"ContainerDied","Data":"fe5555a8ffdf41e38096b81bad9124dd600d99ab90e7c59672a2a2c4194c1415"} Sep 09 16:21:06.450650 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.450312 2562 scope.go:117] "RemoveContainer" containerID="e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3" Sep 09 16:21:06.460722 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.460572 2562 scope.go:117] "RemoveContainer" containerID="f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329" Sep 09 16:21:06.469303 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.469278 2562 scope.go:117] "RemoveContainer" containerID="e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3" Sep 09 16:21:06.469596 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:21:06.469578 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3\": container with ID starting with e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3 not found: ID does not exist" containerID="e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3" Sep 09 16:21:06.469648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.469606 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3"} err="failed to get container status \"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3\": rpc error: code = NotFound desc = could not find container \"e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3\": container with ID starting with e0e47f291b0edc801657effb4fe3c2e61a4b77f335c44a7333acd876f8d1e7b3 not found: ID does not exist" Sep 09 16:21:06.469648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.469623 2562 scope.go:117] "RemoveContainer" containerID="f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329" Sep 09 16:21:06.469867 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:21:06.469849 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329\": container with ID starting with f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329 not found: ID does not exist" containerID="f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329" Sep 09 16:21:06.469904 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.469871 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329"} err="failed to get container status \"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329\": rpc error: code = NotFound desc = could not find container \"f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329\": container with ID starting with f9bf3a6353ec16123bc4a67a199854fa85cf85473eeda17dca56f6df50421329 not found: ID does not exist" Sep 09 16:21:06.473872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.473846 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:21:06.477891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.477869 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-07cc17-openshift-default-8496c8c896-229lr"] Sep 09 16:21:06.855535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.855501 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:21:06.855713 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.855588 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:21:06.867760 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:06.867738 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:21:07.190775 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:07.190695 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" path="/var/lib/kubelet/pods/c01f2d08-2643-4607-8ee6-c34e983d4d8e/volumes" Sep 09 16:21:07.468197 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:07.468110 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:21:11.637468 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:11.637436 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln"] Sep 09 16:21:11.639481 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:11.639460 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0ce96-sk4ln"] Sep 09 16:21:13.189994 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:13.189957 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" path="/var/lib/kubelet/pods/eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2/volumes" Sep 09 16:21:15.333153 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:15.333120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:21:18.290766 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.290729 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z"] Sep 09 16:21:18.291193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291180 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291194 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291203 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291209 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291224 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" containerName="revoke-keys" Sep 09 16:21:18.291234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291229 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" containerName="revoke-keys" Sep 09 16:21:18.291417 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291307 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291417 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291316 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01f2d08-2643-4607-8ee6-c34e983d4d8e" containerName="istio-proxy" Sep 09 16:21:18.291417 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.291322 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaae4d1b-d0b7-46e9-9ca8-e42486d9afe2" containerName="revoke-keys" Sep 09 16:21:18.295980 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.295961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:18.299699 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.299665 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:21:18.299869 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.299666 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:21:18.302127 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.302102 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z"] Sep 09 16:21:18.376594 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.376551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z\" (UID: \"97c74540-fefd-457e-bccf-b4e999bd112b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:18.478196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.478155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z\" (UID: \"97c74540-fefd-457e-bccf-b4e999bd112b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:18.478817 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.478795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z\" (UID: \"97c74540-fefd-457e-bccf-b4e999bd112b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:18.608649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.608612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:18.880805 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:18.880781 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z"] Sep 09 16:21:18.882794 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:21:18.882767 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c74540_fefd_457e_bccf_b4e999bd112b.slice/crio-65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488 WatchSource:0}: Error finding container 65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488: Status 404 returned error can't find the container with id 65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488 Sep 09 16:21:19.546878 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:19.546794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" event={"ID":"97c74540-fefd-457e-bccf-b4e999bd112b","Type":"ContainerStarted","Data":"660c2d12a141423ce82821a739566a85953211a30a83ee0b9a95085b84c3f174"} Sep 09 16:21:19.546878 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:19.546846 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" event={"ID":"97c74540-fefd-457e-bccf-b4e999bd112b","Type":"ContainerStarted","Data":"65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488"} Sep 09 16:21:19.561576 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:19.561530 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" podStartSLOduration=1.561513626 podStartE2EDuration="1.561513626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:19.560836106 +0000 UTC m=+1798.902027936" watchObservedRunningTime="2026-09-09 16:21:19.561513626 +0000 UTC m=+1798.902705457" Sep 09 16:21:20.554300 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:20.554261 2562 generic.go:356] "Generic (PLEG): container finished" podID="97c74540-fefd-457e-bccf-b4e999bd112b" containerID="660c2d12a141423ce82821a739566a85953211a30a83ee0b9a95085b84c3f174" exitCode=0 Sep 09 16:21:20.554300 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:20.554310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" event={"ID":"97c74540-fefd-457e-bccf-b4e999bd112b","Type":"ContainerDied","Data":"660c2d12a141423ce82821a739566a85953211a30a83ee0b9a95085b84c3f174"} Sep 09 16:21:21.699757 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:21.699730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:21.816041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:21.815949 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca\") pod \"97c74540-fefd-457e-bccf-b4e999bd112b\" (UID: \"97c74540-fefd-457e-bccf-b4e999bd112b\") " Sep 09 16:21:21.816384 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:21.816360 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97c74540-fefd-457e-bccf-b4e999bd112b" (UID: "97c74540-fefd-457e-bccf-b4e999bd112b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:21:21.917684 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:21.917648 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c74540-fefd-457e-bccf-b4e999bd112b-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:22.566519 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:22.566480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" event={"ID":"97c74540-fefd-457e-bccf-b4e999bd112b","Type":"ContainerDied","Data":"65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488"} Sep 09 16:21:22.566519 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:22.566509 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z" Sep 09 16:21:22.566726 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:22.566531 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65db836dd351c92441f6d14ef635d5732efab2aba3ef21f48886b25da9241488" Sep 09 16:21:23.761712 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:23.761681 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:21:23.766079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:23.766059 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:21:29.071966 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:29.071926 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z"] Sep 09 16:21:29.075021 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:29.074992 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-008161-vgr6z"] Sep 09 16:21:29.192798 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:29.192764 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c74540-fefd-457e-bccf-b4e999bd112b" path="/var/lib/kubelet/pods/97c74540-fefd-457e-bccf-b4e999bd112b/volumes" Sep 09 16:21:32.231463 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:32.231426 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:21:35.716055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:35.716020 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:21:35.716539 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:35.716296 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" containerID="cri-o://4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938" gracePeriod=30 Sep 09 16:21:40.977142 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:40.977116 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:21:41.017101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017280 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017106 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017280 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017280 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017246 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017320 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trq6b\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017386 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017538 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs\") pod \"bfe613ed-90ca-4586-bcd5-f84311a88788\" (UID: \"bfe613ed-90ca-4586-bcd5-f84311a88788\") " Sep 09 16:21:41.017658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017582 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:21:41.018004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:41.018004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017931 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe613ed-90ca-4586-bcd5-f84311a88788-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.018004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.017949 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.018217 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.018055 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:41.018217 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.018044 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:41.018217 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.018124 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:41.019905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.019879 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b" (OuterVolumeSpecName: "kube-api-access-trq6b") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "kube-api-access-trq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:21:41.019905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.019888 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:21:41.020056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.019975 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:21:41.020467 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.020440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfe613ed-90ca-4586-bcd5-f84311a88788" (UID: "bfe613ed-90ca-4586-bcd5-f84311a88788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:21:41.118922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118889 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.118922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118915 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.118922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118923 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.118922 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118931 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.119188 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118940 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trq6b\" (UniqueName: \"kubernetes.io/projected/bfe613ed-90ca-4586-bcd5-f84311a88788-kube-api-access-trq6b\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.119188 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118948 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.119188 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.118956 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe613ed-90ca-4586-bcd5-f84311a88788-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:21:41.680957 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.680910 2562 generic.go:356] "Generic (PLEG): container finished" podID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerID="4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938" exitCode=0 Sep 09 16:21:41.681152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.680983 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" Sep 09 16:21:41.681152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.681005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerDied","Data":"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938"} Sep 09 16:21:41.681152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.681073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww" event={"ID":"bfe613ed-90ca-4586-bcd5-f84311a88788","Type":"ContainerDied","Data":"26129a4dca807f8d1e235899948bc5f5b3ba3e5b7cabac480ef3519153ea04ff"} Sep 09 16:21:41.681152 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.681091 2562 scope.go:117] "RemoveContainer" containerID="4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938" Sep 09 16:21:41.692121 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.692093 2562 scope.go:117] "RemoveContainer" containerID="6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1" Sep 09 16:21:41.701595 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.701574 2562 scope.go:117] "RemoveContainer" containerID="4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938" Sep 09 16:21:41.701887 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:21:41.701866 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938\": container with ID starting with 4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938 not found: ID does not exist" containerID="4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938" Sep 09 16:21:41.701961 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.701895 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938"} err="failed to get container status \"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938\": rpc error: code = NotFound desc = could not find container \"4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938\": container with ID starting with 4d1514a1837a2a18ae41d2e55ce16bff35271b666b5510a160d5f022279cc938 not found: ID does not exist" Sep 09 16:21:41.701961 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.701913 2562 scope.go:117] "RemoveContainer" containerID="6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1" Sep 09 16:21:41.702177 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:21:41.702162 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1\": container with ID starting with 6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1 not found: ID does not exist" containerID="6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1" Sep 09 16:21:41.702218 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.702181 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1"} err="failed to get container status \"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1\": rpc error: code = NotFound desc = could not find container \"6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1\": container with ID starting with 6e49f611b1482aa1c47bc2a3414326c2a031faecabb7240fc32aff4c01639ab1 not found: ID does not exist" Sep 09 16:21:41.702913 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.702892 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:21:41.708648 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:41.708628 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-008161-openshift-default-7c7fcd86d4-wg9ww"] Sep 09 16:21:43.190708 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:43.190662 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" path="/var/lib/kubelet/pods/bfe613ed-90ca-4586-bcd5-f84311a88788/volumes" Sep 09 16:21:47.030161 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:47.030115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:21:52.445212 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:52.445176 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:21:56.037451 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:21:56.037393 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:22:07.465193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.465149 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerName="main" containerID="cri-o://eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316" gracePeriod=45 Sep 09 16:22:07.469010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.468971 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": EOF" Sep 09 16:22:07.727890 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.727824 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:22:07.777978 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.777937 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.777978 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.777983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7gq2\" (UniqueName: \"kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.778232 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.778232 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778044 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.778232 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778090 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.778232 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778132 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm\") pod \"1ec51227-909f-4bc0-bc9b-7d26781000ad\" (UID: \"1ec51227-909f-4bc0-bc9b-7d26781000ad\") " Sep 09 16:22:07.778465 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778250 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home" (OuterVolumeSpecName: "home") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:22:07.778524 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778479 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:07.778524 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778482 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:22:07.778595 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.778547 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:22:07.780216 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.780190 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm" (OuterVolumeSpecName: "dshm") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:22:07.780334 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.780260 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2" (OuterVolumeSpecName: "kube-api-access-t7gq2") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "kube-api-access-t7gq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:22:07.780430 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.780387 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ec51227-909f-4bc0-bc9b-7d26781000ad" (UID: "1ec51227-909f-4bc0-bc9b-7d26781000ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:22:07.832037 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.832000 2562 generic.go:356] "Generic (PLEG): container finished" podID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerID="eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316" exitCode=0 Sep 09 16:22:07.832196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.832047 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" Sep 09 16:22:07.832196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.832086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" event={"ID":"1ec51227-909f-4bc0-bc9b-7d26781000ad","Type":"ContainerDied","Data":"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316"} Sep 09 16:22:07.832196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.832139 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm" event={"ID":"1ec51227-909f-4bc0-bc9b-7d26781000ad","Type":"ContainerDied","Data":"e45a24eddba502169c51c55b50d2e5fc571d866157f470d1e8e4c31bc5b9fbf6"} Sep 09 16:22:07.832196 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.832159 2562 scope.go:117] "RemoveContainer" containerID="eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316" Sep 09 16:22:07.842101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.842083 2562 scope.go:117] "RemoveContainer" containerID="eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316" Sep 09 16:22:07.842387 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:22:07.842366 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316\": container with ID starting with eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316 not found: ID does not exist" containerID="eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316" Sep 09 16:22:07.842464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.842397 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316"} err="failed to get container status \"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316\": rpc error: code = NotFound desc = could not find container \"eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316\": container with ID starting with eee992166a92e4c845b0becf0f29f1a419b65ed4774bdf94bf7df36c5c5d3316 not found: ID does not exist" Sep 09 16:22:07.853733 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.853701 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:22:07.855769 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.855749 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-13d59f/rate-test-model-13d59f-kserve-cb5cf8c59-dbkgm"] Sep 09 16:22:07.879743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.879717 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:07.879743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.879743 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:07.879909 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.879751 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec51227-909f-4bc0-bc9b-7d26781000ad-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:07.879909 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.879761 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gq2\" (UniqueName: \"kubernetes.io/projected/1ec51227-909f-4bc0-bc9b-7d26781000ad-kube-api-access-t7gq2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:07.879909 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:07.879772 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec51227-909f-4bc0-bc9b-7d26781000ad-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:22:09.189591 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:09.189562 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" path="/var/lib/kubelet/pods/1ec51227-909f-4bc0-bc9b-7d26781000ad/volumes" Sep 09 16:22:09.635314 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:09.635283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:22:22.553798 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.553761 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:22:22.554450 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554391 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.554450 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554449 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554473 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554484 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerName="main" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554491 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerName="main" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554506 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c74540-fefd-457e-bccf-b4e999bd112b" containerName="revoke-keys" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554512 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c74540-fefd-457e-bccf-b4e999bd112b" containerName="revoke-keys" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554589 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec51227-909f-4bc0-bc9b-7d26781000ad" containerName="main" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554601 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c74540-fefd-457e-bccf-b4e999bd112b" containerName="revoke-keys" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554608 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.554637 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.554617 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe613ed-90ca-4586-bcd5-f84311a88788" containerName="istio-proxy" Sep 09 16:22:22.559443 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.559393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.561960 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.561939 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b99260-openshift-default-dockercfg-75txc\"" Sep 09 16:22:22.569079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.569053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:22:22.623026 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.622991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623026 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623240 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkqp\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623240 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623240 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623240 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.623485 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.623320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkqp\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724546 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.724951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.724838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.725065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.725015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.725125 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.725094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.725249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.725228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.726668 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.726639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.727070 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.727051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.731988 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.731968 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkqp\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.732082 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.732019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token\") pod \"e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:22.870910 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:22.870873 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:23.155379 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.155352 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:22:23.157476 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:22:23.157444 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1702248_e44a_4ca7_b2f3_9131380ea979.slice/crio-cff780ea14f55851e3ecadea9d2885b7a425da914b73319c6a23cfed8faf25d1 WatchSource:0}: Error finding container cff780ea14f55851e3ecadea9d2885b7a425da914b73319c6a23cfed8faf25d1: Status 404 returned error can't find the container with id cff780ea14f55851e3ecadea9d2885b7a425da914b73319c6a23cfed8faf25d1 Sep 09 16:22:23.159691 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.159658 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:23.159751 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.159723 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:23.159791 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.159752 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:23.936238 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.936203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerStarted","Data":"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca"} Sep 09 16:22:23.936683 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.936260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerStarted","Data":"cff780ea14f55851e3ecadea9d2885b7a425da914b73319c6a23cfed8faf25d1"} Sep 09 16:22:23.956367 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:23.956313 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podStartSLOduration=1.956298587 podStartE2EDuration="1.956298587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:23.954850213 +0000 UTC m=+1863.296042043" watchObservedRunningTime="2026-09-09 16:22:23.956298587 +0000 UTC m=+1863.297490415" Sep 09 16:22:24.871870 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:24.871834 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:24.876695 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:24.876668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:24.942095 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:24.942070 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:24.943165 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:24.943144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:27.121785 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.121752 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:22:27.125132 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.125109 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.128862 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.128837 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 16:22:27.129000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.128838 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n2qxg\"" Sep 09 16:22:27.129000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.128907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:22:27.129000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.128916 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b99260-serving-cert\"" Sep 09 16:22:27.129000 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.128931 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 16:22:27.135978 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.135524 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:22:27.271041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.271003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkjb\" (UniqueName: \"kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.271233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.271049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.271233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.271202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.271354 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.271247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.372829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.372727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.372829 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.372783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.373065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.372853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkjb\" (UniqueName: \"kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.373065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.372886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.373448 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.373397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.375238 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.375213 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.375338 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.375323 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.381641 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.381620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkjb\" (UniqueName: \"kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb\") pod \"maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.441705 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.441665 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:27.718663 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.718638 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:22:27.720208 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:22:27.720182 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8271239_edb1_47e2_8e79_cfbe157b3502.slice/crio-be3eb3299e328b2d67a325426b17ffddaea702069efd4131486480c9fcd8fc27 WatchSource:0}: Error finding container be3eb3299e328b2d67a325426b17ffddaea702069efd4131486480c9fcd8fc27: Status 404 returned error can't find the container with id be3eb3299e328b2d67a325426b17ffddaea702069efd4131486480c9fcd8fc27 Sep 09 16:22:27.721860 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.721839 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 16:22:27.961785 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:27.961741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" event={"ID":"d8271239-edb1-47e2-8e79-cfbe157b3502","Type":"ContainerStarted","Data":"be3eb3299e328b2d67a325426b17ffddaea702069efd4131486480c9fcd8fc27"} Sep 09 16:22:30.982876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:30.982832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" event={"ID":"d8271239-edb1-47e2-8e79-cfbe157b3502","Type":"ContainerStarted","Data":"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5"} Sep 09 16:22:30.983249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:30.982970 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:31.005062 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:31.005016 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" podStartSLOduration=1.703235992 podStartE2EDuration="4.005002623s" totalImagesPullingTime="2.301766631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:22:31.002113667 +0000 UTC m=+1870.343305497" watchObservedRunningTime="2026-09-09 16:22:31.005002623 +0000 UTC m=+1870.346194452" Sep 09 16:22:33.350277 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:33.350240 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:22:36.999770 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:36.999717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:22:44.065078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.065042 2562 generic.go:356] "Generic (PLEG): container finished" podID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerID="cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca" exitCode=0 Sep 09 16:22:44.065529 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.065109 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerDied","Data":"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca"} Sep 09 16:22:44.065575 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.065540 2562 scope.go:117] "RemoveContainer" containerID="cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca" Sep 09 16:22:44.066673 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.066638 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:44.066736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.066711 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:44.066776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:44.066747 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 16:22:45.072818 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:45.072782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerStarted","Data":"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37"} Sep 09 16:22:45.871488 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:45.871443 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:45.871711 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:45.871689 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 09 16:22:45.871786 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:45.871733 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 09 16:22:46.872101 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:46.872064 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 09 16:22:46.872488 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:46.872123 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 09 16:22:47.872079 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:47.872038 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 09 16:22:47.872291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:47.872109 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 09 16:22:48.871786 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:48.871746 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 09 16:22:48.871937 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:48.871811 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 09 16:22:49.871891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:49.871734 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 09 16:22:49.871891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:49.871809 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 09 16:22:50.876302 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:50.876270 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:50.876780 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:50.876604 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:22:50.877205 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:22:50.877187 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:23:08.334349 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:08.334314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:23:21.545042 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:21.545006 2562 scope.go:117] "RemoveContainer" containerID="c088afb8ffdd689708f4b1060fdb1cda12e2ce9adf77da11d8f1809a8356f66e" Sep 09 16:23:30.629130 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:30.629096 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:23:41.365214 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.365177 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:23:41.369164 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.369142 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.371555 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.371535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-6ddd57-serving-cert\"" Sep 09 16:23:41.380644 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.380432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:23:41.436061 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.436031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.436208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.436081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.436208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.436106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqtj\" (UniqueName: \"kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.436208 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.436161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.536784 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.536747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.536967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.536792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqtj\" (UniqueName: \"kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.536967 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.536835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.537071 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.536975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.537634 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.537605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.539291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.539267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.539381 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.539301 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.544770 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.544748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqtj\" (UniqueName: \"kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj\") pod \"maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.682043 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.681966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:41.964191 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:41.964167 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:23:41.966017 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:23:41.965991 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae18204_dbfc_4e67_91e7_82ffbff494ed.slice/crio-712c109e78957e335f900a8b5a3e4a5e9aa67906e0425373f4303697ff8d4cd7 WatchSource:0}: Error finding container 712c109e78957e335f900a8b5a3e4a5e9aa67906e0425373f4303697ff8d4cd7: Status 404 returned error can't find the container with id 712c109e78957e335f900a8b5a3e4a5e9aa67906e0425373f4303697ff8d4cd7 Sep 09 16:23:42.414815 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:42.414781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" event={"ID":"1ae18204-dbfc-4e67-91e7-82ffbff494ed","Type":"ContainerStarted","Data":"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba"} Sep 09 16:23:42.415180 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:42.414844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" event={"ID":"1ae18204-dbfc-4e67-91e7-82ffbff494ed","Type":"ContainerStarted","Data":"712c109e78957e335f900a8b5a3e4a5e9aa67906e0425373f4303697ff8d4cd7"} Sep 09 16:23:42.415180 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:42.414942 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:42.431848 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:42.431798 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" podStartSLOduration=1.049129171 podStartE2EDuration="1.431783664s" totalImagesPullingTime="382.654493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:42.430862498 +0000 UTC m=+1941.772054328" watchObservedRunningTime="2026-09-09 16:23:42.431783664 +0000 UTC m=+1941.772975493" Sep 09 16:23:48.425297 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:48.425265 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:23:50.564462 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:23:50.564427 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:24:12.009244 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.009203 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:24:12.009882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.009425 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" podUID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" containerName="maas-api" containerID="cri-o://e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba" gracePeriod=30 Sep 09 16:24:12.265725 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.265656 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:24:12.348036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.347999 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnqtj\" (UniqueName: \"kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj\") pod \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " Sep 09 16:24:12.348036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.348037 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca\") pod \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " Sep 09 16:24:12.348252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.348097 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls\") pod \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " Sep 09 16:24:12.348252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.348148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs\") pod \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\" (UID: \"1ae18204-dbfc-4e67-91e7-82ffbff494ed\") " Sep 09 16:24:12.348472 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.348445 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ae18204-dbfc-4e67-91e7-82ffbff494ed" (UID: "1ae18204-dbfc-4e67-91e7-82ffbff494ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:24:12.348610 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.348591 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae18204-dbfc-4e67-91e7-82ffbff494ed-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:24:12.350275 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.350243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ae18204-dbfc-4e67-91e7-82ffbff494ed" (UID: "1ae18204-dbfc-4e67-91e7-82ffbff494ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:24:12.350384 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.350307 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj" (OuterVolumeSpecName: "kube-api-access-hnqtj") pod "1ae18204-dbfc-4e67-91e7-82ffbff494ed" (UID: "1ae18204-dbfc-4e67-91e7-82ffbff494ed"). InnerVolumeSpecName "kube-api-access-hnqtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:24:12.350608 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.350582 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ae18204-dbfc-4e67-91e7-82ffbff494ed" (UID: "1ae18204-dbfc-4e67-91e7-82ffbff494ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:24:12.449456 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.449380 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqtj\" (UniqueName: \"kubernetes.io/projected/1ae18204-dbfc-4e67-91e7-82ffbff494ed-kube-api-access-hnqtj\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:24:12.449456 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.449451 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:24:12.449456 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.449469 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae18204-dbfc-4e67-91e7-82ffbff494ed-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:24:12.591320 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.591225 2562 generic.go:356] "Generic (PLEG): container finished" podID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" containerID="e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba" exitCode=0 Sep 09 16:24:12.591612 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.591590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" event={"ID":"1ae18204-dbfc-4e67-91e7-82ffbff494ed","Type":"ContainerDied","Data":"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba"} Sep 09 16:24:12.591727 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.591697 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" Sep 09 16:24:12.592056 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.591778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg" event={"ID":"1ae18204-dbfc-4e67-91e7-82ffbff494ed","Type":"ContainerDied","Data":"712c109e78957e335f900a8b5a3e4a5e9aa67906e0425373f4303697ff8d4cd7"} Sep 09 16:24:12.592168 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.591794 2562 scope.go:117] "RemoveContainer" containerID="e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba" Sep 09 16:24:12.602271 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.602251 2562 scope.go:117] "RemoveContainer" containerID="e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba" Sep 09 16:24:12.602640 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:24:12.602603 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba\": container with ID starting with e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba not found: ID does not exist" containerID="e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba" Sep 09 16:24:12.602640 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.602634 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba"} err="failed to get container status \"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba\": rpc error: code = NotFound desc = could not find container \"e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba\": container with ID starting with e5bc469eb13966746ed5b089319c6a7b642382cee3d7f0cdd14aeb1ba840d0ba not found: ID does not exist" Sep 09 16:24:12.626193 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.626163 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:24:12.630373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:12.630349 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6ddd57-759497d9db-cnjjg"] Sep 09 16:24:13.191366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:13.191334 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" path="/var/lib/kubelet/pods/1ae18204-dbfc-4e67-91e7-82ffbff494ed/volumes" Sep 09 16:24:16.138122 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:16.138083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:24:35.536830 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:35.536789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:24:48.029103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:24:48.029064 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:25:37.236535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.236499 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:25:37.237568 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.237529 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" podUID="d8271239-edb1-47e2-8e79-cfbe157b3502" containerName="maas-api" containerID="cri-o://7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5" gracePeriod=30 Sep 09 16:25:37.489319 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.489264 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:25:37.567776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.567748 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs\") pod \"d8271239-edb1-47e2-8e79-cfbe157b3502\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " Sep 09 16:25:37.567923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.567848 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls\") pod \"d8271239-edb1-47e2-8e79-cfbe157b3502\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " Sep 09 16:25:37.567923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.567888 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca\") pod \"d8271239-edb1-47e2-8e79-cfbe157b3502\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " Sep 09 16:25:37.567923 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.567914 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwkjb\" (UniqueName: \"kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb\") pod \"d8271239-edb1-47e2-8e79-cfbe157b3502\" (UID: \"d8271239-edb1-47e2-8e79-cfbe157b3502\") " Sep 09 16:25:37.568235 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.568210 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8271239-edb1-47e2-8e79-cfbe157b3502" (UID: "d8271239-edb1-47e2-8e79-cfbe157b3502"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:25:37.569980 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.569955 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8271239-edb1-47e2-8e79-cfbe157b3502" (UID: "d8271239-edb1-47e2-8e79-cfbe157b3502"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:25:37.570113 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.569979 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8271239-edb1-47e2-8e79-cfbe157b3502" (UID: "d8271239-edb1-47e2-8e79-cfbe157b3502"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:25:37.570113 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.570087 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb" (OuterVolumeSpecName: "kube-api-access-fwkjb") pod "d8271239-edb1-47e2-8e79-cfbe157b3502" (UID: "d8271239-edb1-47e2-8e79-cfbe157b3502"). InnerVolumeSpecName "kube-api-access-fwkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:25:37.669104 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.669069 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:37.669104 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.669101 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8271239-edb1-47e2-8e79-cfbe157b3502-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:37.669104 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.669111 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8271239-edb1-47e2-8e79-cfbe157b3502-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:37.669347 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:37.669121 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkjb\" (UniqueName: \"kubernetes.io/projected/d8271239-edb1-47e2-8e79-cfbe157b3502-kube-api-access-fwkjb\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:38.092252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.092213 2562 generic.go:356] "Generic (PLEG): container finished" podID="d8271239-edb1-47e2-8e79-cfbe157b3502" containerID="7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5" exitCode=0 Sep 09 16:25:38.092252 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.092246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" event={"ID":"d8271239-edb1-47e2-8e79-cfbe157b3502","Type":"ContainerDied","Data":"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5"} Sep 09 16:25:38.092565 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.092255 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" Sep 09 16:25:38.092565 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.092306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v" event={"ID":"d8271239-edb1-47e2-8e79-cfbe157b3502","Type":"ContainerDied","Data":"be3eb3299e328b2d67a325426b17ffddaea702069efd4131486480c9fcd8fc27"} Sep 09 16:25:38.092565 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.092337 2562 scope.go:117] "RemoveContainer" containerID="7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5" Sep 09 16:25:38.102041 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.102023 2562 scope.go:117] "RemoveContainer" containerID="7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5" Sep 09 16:25:38.102300 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:25:38.102278 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5\": container with ID starting with 7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5 not found: ID does not exist" containerID="7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5" Sep 09 16:25:38.102337 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.102309 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5"} err="failed to get container status \"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5\": rpc error: code = NotFound desc = could not find container \"7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5\": container with ID starting with 7f9645dc7f5680b98760778520adb7fd25145c1783d500443a1559e972f770c5 not found: ID does not exist" Sep 09 16:25:38.114258 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.114228 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:25:38.117326 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:38.117306 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b99260-677b66d7fd-pjl7v"] Sep 09 16:25:39.190145 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:39.190111 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8271239-edb1-47e2-8e79-cfbe157b3502" path="/var/lib/kubelet/pods/d8271239-edb1-47e2-8e79-cfbe157b3502/volumes" Sep 09 16:25:40.429676 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:40.429646 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:25:43.779556 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:43.779455 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:25:43.779995 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:43.779836 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" containerID="cri-o://a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37" gracePeriod=30 Sep 09 16:25:49.033479 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.033453 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:25:49.156897 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.156859 2562 generic.go:356] "Generic (PLEG): container finished" podID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerID="a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37" exitCode=0 Sep 09 16:25:49.157078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.156914 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" Sep 09 16:25:49.157078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.156936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerDied","Data":"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37"} Sep 09 16:25:49.157078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.157010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt" event={"ID":"d1702248-e44a-4ca7-b2f3-9131380ea979","Type":"ContainerDied","Data":"cff780ea14f55851e3ecadea9d2885b7a425da914b73319c6a23cfed8faf25d1"} Sep 09 16:25:49.157078 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.157040 2562 scope.go:117] "RemoveContainer" containerID="a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37" Sep 09 16:25:49.165999 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.165980 2562 scope.go:117] "RemoveContainer" containerID="cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca" Sep 09 16:25:49.174065 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.174047 2562 scope.go:117] "RemoveContainer" containerID="a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37" Sep 09 16:25:49.174350 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:25:49.174331 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37\": container with ID starting with a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37 not found: ID does not exist" containerID="a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37" Sep 09 16:25:49.174433 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.174359 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37"} err="failed to get container status \"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37\": rpc error: code = NotFound desc = could not find container \"a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37\": container with ID starting with a8094391f68ee02d4e86100efe0a846916235ee6208165a059b1b94b5f544f37 not found: ID does not exist" Sep 09 16:25:49.174433 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.174376 2562 scope.go:117] "RemoveContainer" containerID="cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca" Sep 09 16:25:49.174680 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:25:49.174661 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca\": container with ID starting with cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca not found: ID does not exist" containerID="cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca" Sep 09 16:25:49.174730 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.174685 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca"} err="failed to get container status \"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca\": rpc error: code = NotFound desc = could not find container \"cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca\": container with ID starting with cc531e01b1737c811da2b054ee072d4e52b22b253fad3e792bebe656b8e09eca not found: ID does not exist" Sep 09 16:25:49.180987 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.180970 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.180993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181055 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181012 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfkqp\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181055 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181129 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181171 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket\") pod \"d1702248-e44a-4ca7-b2f3-9131380ea979\" (UID: \"d1702248-e44a-4ca7-b2f3-9131380ea979\") " Sep 09 16:25:49.181445 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181383 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:25:49.181445 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181404 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:25:49.181573 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181523 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:25:49.181658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181618 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:25:49.181658 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181640 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:25:49.181795 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181777 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1702248-e44a-4ca7-b2f3-9131380ea979-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.181832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181801 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.181832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181817 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.181917 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181831 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.181917 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.181844 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.183133 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.183092 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:25:49.183268 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.183246 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp" (OuterVolumeSpecName: "kube-api-access-mfkqp") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "kube-api-access-mfkqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:25:49.183557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.183526 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:25:49.183659 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.183577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1702248-e44a-4ca7-b2f3-9131380ea979" (UID: "d1702248-e44a-4ca7-b2f3-9131380ea979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:25:49.282537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.282460 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.282537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.282489 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.282537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.282503 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkqp\" (UniqueName: \"kubernetes.io/projected/d1702248-e44a-4ca7-b2f3-9131380ea979-kube-api-access-mfkqp\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.282537 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.282515 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1702248-e44a-4ca7-b2f3-9131380ea979-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:25:49.474424 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.474384 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:25:49.480782 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:49.480743 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b99260-openshift-default-cbd869448-pd6gt"] Sep 09 16:25:51.189747 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:51.189717 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" path="/var/lib/kubelet/pods/d1702248-e44a-4ca7-b2f3-9131380ea979/volumes" Sep 09 16:25:54.823154 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:54.823119 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5"] Sep 09 16:25:54.829832 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:54.829802 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-131689-29816175-cn2m5"] Sep 09 16:25:55.193803 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:55.193456 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1ef53e-42e1-42d2-8cdb-e322f46b51f3" path="/var/lib/kubelet/pods/fb1ef53e-42e1-42d2-8cdb-e322f46b51f3/volumes" Sep 09 16:25:57.830816 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:25:57.830776 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:26:01.332340 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.332308 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:26:01.332846 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.332585 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" containerID="cri-o://70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb" gracePeriod=30 Sep 09 16:26:01.685323 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685288 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd"] Sep 09 16:26:01.685776 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685762 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:26:01.685825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685778 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:26:01.685825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685793 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:26:01.685825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685799 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:26:01.685825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685811 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8271239-edb1-47e2-8e79-cfbe157b3502" containerName="maas-api" Sep 09 16:26:01.685825 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685817 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8271239-edb1-47e2-8e79-cfbe157b3502" containerName="maas-api" Sep 09 16:26:01.685983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685835 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" containerName="maas-api" Sep 09 16:26:01.685983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685840 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" containerName="maas-api" Sep 09 16:26:01.685983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685899 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:26:01.685983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685907 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae18204-dbfc-4e67-91e7-82ffbff494ed" containerName="maas-api" Sep 09 16:26:01.685983 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.685916 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8271239-edb1-47e2-8e79-cfbe157b3502" containerName="maas-api" Sep 09 16:26:01.688796 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.688780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:01.691441 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.691401 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:26:01.691572 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.691461 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgl2h\"" Sep 09 16:26:01.695052 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.695028 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd"] Sep 09 16:26:01.798362 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.798330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd\" (UID: \"57786343-206f-49c3-b6c1-d3d0dd004a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:01.899716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.899679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd\" (UID: \"57786343-206f-49c3-b6c1-d3d0dd004a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:01.900366 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:01.900345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd\" (UID: \"57786343-206f-49c3-b6c1-d3d0dd004a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:02.001375 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:02.001285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:02.274257 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:02.274233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd"] Sep 09 16:26:02.275687 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:26:02.275658 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57786343_206f_49c3_b6c1_d3d0dd004a47.slice/crio-8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7 WatchSource:0}: Error finding container 8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7: Status 404 returned error can't find the container with id 8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7 Sep 09 16:26:03.242018 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:03.241933 2562 generic.go:356] "Generic (PLEG): container finished" podID="57786343-206f-49c3-b6c1-d3d0dd004a47" containerID="314116488d8af19cee8963e6f0e5a0bb92d74f2d425f888bbddc5afcb2c11505" exitCode=0 Sep 09 16:26:03.242391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:03.242023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" event={"ID":"57786343-206f-49c3-b6c1-d3d0dd004a47","Type":"ContainerDied","Data":"314116488d8af19cee8963e6f0e5a0bb92d74f2d425f888bbddc5afcb2c11505"} Sep 09 16:26:03.242391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:03.242079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" event={"ID":"57786343-206f-49c3-b6c1-d3d0dd004a47","Type":"ContainerStarted","Data":"8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7"} Sep 09 16:26:04.385796 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:04.385766 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:04.427915 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:04.427876 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca\") pod \"57786343-206f-49c3-b6c1-d3d0dd004a47\" (UID: \"57786343-206f-49c3-b6c1-d3d0dd004a47\") " Sep 09 16:26:04.428246 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:04.428222 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57786343-206f-49c3-b6c1-d3d0dd004a47" (UID: "57786343-206f-49c3-b6c1-d3d0dd004a47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:26:04.529067 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:04.528981 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57786343-206f-49c3-b6c1-d3d0dd004a47-maas-api-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:05.254261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:05.254228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" event={"ID":"57786343-206f-49c3-b6c1-d3d0dd004a47","Type":"ContainerDied","Data":"8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7"} Sep 09 16:26:05.254261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:05.254251 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd" Sep 09 16:26:05.254508 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:05.254279 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cedc5b1eb62260750fe6b77dfc0acb069da9afb3c25791465631db5c74b04c7" Sep 09 16:26:06.594125 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.594102 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:26:06.649649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649584 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.649649 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649624 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.649813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649651 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlbcz\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.649813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649684 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.649813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.649813 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649785 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.650010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649814 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.650010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649865 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.650010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649867 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data" (OuterVolumeSpecName: "istio-data") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:06.650010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649882 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:06.650010 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.649902 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo\") pod \"77d24225-9712-457c-ae5a-252aad07dff1\" (UID: \"77d24225-9712-457c-ae5a-252aad07dff1\") " Sep 09 16:26:06.650243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650095 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:06.650243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650108 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:26:06.650243 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:06.650350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650287 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.650350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650304 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d24225-9712-457c-ae5a-252aad07dff1-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.650350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650318 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.650350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650331 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.650350 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.650343 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.652013 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.651992 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:26:06.652090 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.652029 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:06.652278 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.652261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz" (OuterVolumeSpecName: "kube-api-access-wlbcz") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "kube-api-access-wlbcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:26:06.652322 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.652290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token" (OuterVolumeSpecName: "istio-token") pod "77d24225-9712-457c-ae5a-252aad07dff1" (UID: "77d24225-9712-457c-ae5a-252aad07dff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:26:06.751309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.751262 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d24225-9712-457c-ae5a-252aad07dff1-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.751309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.751303 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d24225-9712-457c-ae5a-252aad07dff1-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.751309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.751313 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbcz\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-kube-api-access-wlbcz\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:06.751309 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:06.751322 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d24225-9712-457c-ae5a-252aad07dff1-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:07.266165 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.266128 2562 generic.go:356] "Generic (PLEG): container finished" podID="77d24225-9712-457c-ae5a-252aad07dff1" containerID="70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb" exitCode=0 Sep 09 16:26:07.266348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.266176 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" Sep 09 16:26:07.266348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.266216 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerDied","Data":"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb"} Sep 09 16:26:07.266348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.266277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685" event={"ID":"77d24225-9712-457c-ae5a-252aad07dff1","Type":"ContainerDied","Data":"bdbcc4f93131d8979344c69e76a107d6a2a5b1ecad574b8b61c18b4551c296dc"} Sep 09 16:26:07.266348 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.266304 2562 scope.go:117] "RemoveContainer" containerID="70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb" Sep 09 16:26:07.275295 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.275274 2562 scope.go:117] "RemoveContainer" containerID="5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c" Sep 09 16:26:07.283535 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.283515 2562 scope.go:117] "RemoveContainer" containerID="70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb" Sep 09 16:26:07.283856 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:26:07.283826 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb\": container with ID starting with 70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb not found: ID does not exist" containerID="70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb" Sep 09 16:26:07.283960 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.283857 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb"} err="failed to get container status \"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb\": rpc error: code = NotFound desc = could not find container \"70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb\": container with ID starting with 70611d88a8ba660f2e6780fd74b00aad4c5a995a66394b1295dd2ba2000f24bb not found: ID does not exist" Sep 09 16:26:07.283960 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.283875 2562 scope.go:117] "RemoveContainer" containerID="5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c" Sep 09 16:26:07.284122 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:26:07.284105 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c\": container with ID starting with 5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c not found: ID does not exist" containerID="5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c" Sep 09 16:26:07.284174 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.284129 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c"} err="failed to get container status \"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c\": rpc error: code = NotFound desc = could not find container \"5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c\": container with ID starting with 5e329007cc8243e797e206777715e5978c39c2096aa6cc6c80fb3907ab4d0a7c not found: ID does not exist" Sep 09 16:26:07.285489 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.285470 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:26:07.289333 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:07.289308 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-131689-openshift-default-6b67897f44-29685"] Sep 09 16:26:09.189920 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:09.189881 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d24225-9712-457c-ae5a-252aad07dff1" path="/var/lib/kubelet/pods/77d24225-9712-457c-ae5a-252aad07dff1/volumes" Sep 09 16:26:12.147510 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:12.147475 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx"] Sep 09 16:26:12.153824 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:12.153795 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-13d59f-29816175-gjgcx"] Sep 09 16:26:12.333029 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:12.332997 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd"] Sep 09 16:26:12.336364 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:12.336337 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-13d59f-5rbbd"] Sep 09 16:26:13.190654 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:13.190614 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064ff2f7-fea2-42d8-ae69-0fe4a576e4e9" path="/var/lib/kubelet/pods/064ff2f7-fea2-42d8-ae69-0fe4a576e4e9/volumes" Sep 09 16:26:13.191025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:13.190931 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57786343-206f-49c3-b6c1-d3d0dd004a47" path="/var/lib/kubelet/pods/57786343-206f-49c3-b6c1-d3d0dd004a47/volumes" Sep 09 16:26:14.837738 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:14.837701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:26:18.757323 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:18.757285 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:26:18.757745 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:18.757563 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" containerID="cri-o://938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9" gracePeriod=30 Sep 09 16:26:21.588284 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:21.588250 2562 scope.go:117] "RemoveContainer" containerID="8dd6b765328266f4fdb2303d542da6f0a05787eef494252b1f4c6745e4daac48" Sep 09 16:26:21.598251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:21.598223 2562 scope.go:117] "RemoveContainer" containerID="1aa96d4726c875173fe5125b83fff227e6d105a45de5afec8ad6307a68e20343" Sep 09 16:26:21.606877 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:21.606856 2562 scope.go:117] "RemoveContainer" containerID="06c933e9be532585a5e83865dee2e83dfc83b55fb8df28b07f774b5c9cf3c134" Sep 09 16:26:23.132318 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:23.132283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:26:23.799944 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:23.799921 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:26:23.807004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:23.806977 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:26:24.031643 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.031577 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:26:24.123583 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.123535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.123792 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.123616 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.123792 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.123682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124032 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.123972 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124032 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.123994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:24.124275 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124082 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:26:24.124275 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124043 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124466 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124216 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:24.124466 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124466 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124396 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124466 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124461 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124722 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124499 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgddd\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd\") pod \"16c26e17-7004-40e3-8d24-30437b9b077d\" (UID: \"16c26e17-7004-40e3-8d24-30437b9b077d\") " Sep 09 16:26:24.124992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124866 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-credential-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.124992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124889 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-socket\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.124992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124905 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c26e17-7004-40e3-8d24-30437b9b077d-istiod-ca-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.124992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.124914 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data" (OuterVolumeSpecName: "istio-data") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:24.125363 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.125328 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:24.126390 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.126356 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 16:26:24.126536 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.126511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 16:26:24.127073 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.127045 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token" (OuterVolumeSpecName: "istio-token") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:26:24.127265 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.127244 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd" (OuterVolumeSpecName: "kube-api-access-mgddd") pod "16c26e17-7004-40e3-8d24-30437b9b077d" (UID: "16c26e17-7004-40e3-8d24-30437b9b077d"). InnerVolumeSpecName "kube-api-access-mgddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:26:24.225868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225821 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-envoy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.225868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225861 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c26e17-7004-40e3-8d24-30437b9b077d-istio-podinfo\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.225868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225876 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-istio-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.226486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225888 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-istio-data\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.226486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225901 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c26e17-7004-40e3-8d24-30437b9b077d-workload-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.226486 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.225916 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgddd\" (UniqueName: \"kubernetes.io/projected/16c26e17-7004-40e3-8d24-30437b9b077d-kube-api-access-mgddd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:26:24.363476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.363404 2562 generic.go:356] "Generic (PLEG): container finished" podID="16c26e17-7004-40e3-8d24-30437b9b077d" containerID="938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9" exitCode=0 Sep 09 16:26:24.363476 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.363475 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" Sep 09 16:26:24.363675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.363505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerDied","Data":"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9"} Sep 09 16:26:24.363675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.363557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6" event={"ID":"16c26e17-7004-40e3-8d24-30437b9b077d","Type":"ContainerDied","Data":"69df6d773633de4867672b286820328fb83150731a745f7a14540d3b5fba509f"} Sep 09 16:26:24.363675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.363579 2562 scope.go:117] "RemoveContainer" containerID="938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9" Sep 09 16:26:24.373483 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.373458 2562 scope.go:117] "RemoveContainer" containerID="83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d" Sep 09 16:26:24.381905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.381887 2562 scope.go:117] "RemoveContainer" containerID="938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9" Sep 09 16:26:24.382228 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:26:24.382199 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9\": container with ID starting with 938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9 not found: ID does not exist" containerID="938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9" Sep 09 16:26:24.382319 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.382231 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9"} err="failed to get container status \"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9\": rpc error: code = NotFound desc = could not find container \"938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9\": container with ID starting with 938aa3f3a7ede20191fe226baca4fabc2bf7167d48fb835a49a9cf06b59aa5e9 not found: ID does not exist" Sep 09 16:26:24.382319 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.382254 2562 scope.go:117] "RemoveContainer" containerID="83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d" Sep 09 16:26:24.382758 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:26:24.382598 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d\": container with ID starting with 83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d not found: ID does not exist" containerID="83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d" Sep 09 16:26:24.382758 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.382635 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d"} err="failed to get container status \"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d\": rpc error: code = NotFound desc = could not find container \"83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d\": container with ID starting with 83ffb58d521b34483033afba30527fc6c0640ece100c19ff1d58f1eaf26e2d5d not found: ID does not exist" Sep 09 16:26:24.385344 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.385322 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:26:24.388468 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:24.388446 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-13d59f-openshift-default-84c66846d6-z7fh6"] Sep 09 16:26:25.189919 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:26:25.189881 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" path="/var/lib/kubelet/pods/16c26e17-7004-40e3-8d24-30437b9b077d/volumes" Sep 09 16:27:21.647961 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:27:21.647877 2562 scope.go:117] "RemoveContainer" containerID="660c2d12a141423ce82821a739566a85953211a30a83ee0b9a95085b84c3f174" Sep 09 16:27:21.658959 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:27:21.658934 2562 scope.go:117] "RemoveContainer" containerID="aee8ab186996348f66c2ff70dc2599a4b7ecfede0d021292f0d5a5bacd925981" Sep 09 16:27:21.666964 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:27:21.666945 2562 scope.go:117] "RemoveContainer" containerID="3eabab11d4371cf3c9db80949272d7c3b26e84e5af2838ae0c0129bf6ff7303b" Sep 09 16:27:45.440521 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:27:45.440473 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:28:12.733883 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:28:12.733846 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:28:19.852835 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:28:19.852798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:28:25.934961 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:28:25.934922 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:28:31.534930 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:28:31.534892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:28:50.629014 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:28:50.628976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:29:00.934230 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:29:00.934189 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:30:30.529237 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:30:30.529152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:30:38.622946 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:30:38.622911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:30:44.528897 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:30:44.528842 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:30:57.436383 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:30:57.436345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:31:20.133812 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:20.133775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:31:23.842709 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:23.842680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:31:23.850564 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:23.850538 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:31:38.534595 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:38.534553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:31:46.741186 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:46.741095 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:31:56.129426 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.129381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:31:56.447517 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447430 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:31:56.447868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447852 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447870 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447885 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447891 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447909 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447914 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447924 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57786343-206f-49c3-b6c1-d3d0dd004a47" containerName="revoke-keys" Sep 09 16:31:56.447970 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.447931 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57786343-206f-49c3-b6c1-d3d0dd004a47" containerName="revoke-keys" Sep 09 16:31:56.448261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.448041 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:31:56.448261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.448052 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57786343-206f-49c3-b6c1-d3d0dd004a47" containerName="revoke-keys" Sep 09 16:31:56.448261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.448060 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:31:56.448261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.448069 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:31:56.448261 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.448081 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1702248-e44a-4ca7-b2f3-9131380ea979" containerName="istio-proxy" Sep 09 16:31:56.451597 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.451576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.455962 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.455933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 16:31:56.456086 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.455992 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 16:31:56.456086 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.456013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n2qxg\"" Sep 09 16:31:56.456340 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.456322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 16:31:56.456443 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.456356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 16:31:56.461386 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.461365 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:31:56.538279 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.538247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.538472 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.538321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.538472 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.538451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7stm\" (UniqueName: \"kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.538550 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.538480 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.639510 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.639473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7stm\" (UniqueName: \"kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.639510 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.639509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.639743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.639544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.639743 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.639639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.640256 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.640235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.642040 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.642017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.642289 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.642070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.647626 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.647609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7stm\" (UniqueName: \"kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm\") pod \"maas-api-6b4b89c5b5-sf4mj\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:56.763505 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:56.763421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:57.035982 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:57.035957 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:31:57.037969 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:31:57.037934 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5735c48_4c59_464d_af89_0f08c73bc1c4.slice/crio-bc8be494dd4a9b8053b0f4c16b57617aa54622edb14abf401f5275c15731e640 WatchSource:0}: Error finding container bc8be494dd4a9b8053b0f4c16b57617aa54622edb14abf401f5275c15731e640: Status 404 returned error can't find the container with id bc8be494dd4a9b8053b0f4c16b57617aa54622edb14abf401f5275c15731e640 Sep 09 16:31:57.039383 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:57.039362 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 16:31:57.102036 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:57.101998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" event={"ID":"a5735c48-4c59-464d-af89-0f08c73bc1c4","Type":"ContainerStarted","Data":"bc8be494dd4a9b8053b0f4c16b57617aa54622edb14abf401f5275c15731e640"} Sep 09 16:31:58.109043 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:58.109002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" event={"ID":"a5735c48-4c59-464d-af89-0f08c73bc1c4","Type":"ContainerStarted","Data":"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb"} Sep 09 16:31:58.109498 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:58.109095 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:31:58.126574 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:31:58.126529 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" podStartSLOduration=1.64416929 podStartE2EDuration="2.12651318s" totalImagesPullingTime="482.34389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:31:58.126096898 +0000 UTC m=+2437.467288725" watchObservedRunningTime="2026-09-09 16:31:58.12651318 +0000 UTC m=+2437.467705006" Sep 09 16:32:04.119116 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.119077 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:32:04.123696 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.123671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:32:04.244866 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.244825 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8585bfb869-j668k"] Sep 09 16:32:04.245452 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.245429 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:32:04.245452 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.245452 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c26e17-7004-40e3-8d24-30437b9b077d" containerName="istio-proxy" Sep 09 16:32:04.245652 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.245552 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d24225-9712-457c-ae5a-252aad07dff1" containerName="istio-proxy" Sep 09 16:32:04.248755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.248736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.259027 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.259001 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8585bfb869-j668k"] Sep 09 16:32:04.415103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.415021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6qw\" (UniqueName: \"kubernetes.io/projected/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-kube-api-access-jq6qw\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.415103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.415055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-metrics-certs\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.415103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.415092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-openshift-service-ca\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.415330 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.415138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-maas-api-tls\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.515897 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.515858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6qw\" (UniqueName: \"kubernetes.io/projected/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-kube-api-access-jq6qw\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.515897 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.515896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-metrics-certs\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.516125 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.516016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-openshift-service-ca\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.516125 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.516065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-maas-api-tls\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.516756 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.516722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-openshift-service-ca\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.518457 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.518438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-maas-api-tls\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.518557 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.518474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-metrics-certs\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.523565 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.523540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6qw\" (UniqueName: \"kubernetes.io/projected/f78cdc1e-55e8-487b-bdb3-ca06eeefb5df-kube-api-access-jq6qw\") pod \"maas-api-8585bfb869-j668k\" (UID: \"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df\") " pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.561447 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.561402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:04.831868 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:04.831839 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8585bfb869-j668k"] Sep 09 16:32:04.833686 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:32:04.833656 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78cdc1e_55e8_487b_bdb3_ca06eeefb5df.slice/crio-934568e26bb54cbb75815314e323dcc737dca890e09333246c02b2fc7c945487 WatchSource:0}: Error finding container 934568e26bb54cbb75815314e323dcc737dca890e09333246c02b2fc7c945487: Status 404 returned error can't find the container with id 934568e26bb54cbb75815314e323dcc737dca890e09333246c02b2fc7c945487 Sep 09 16:32:05.149484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:05.149441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" event={"ID":"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df","Type":"ContainerStarted","Data":"934568e26bb54cbb75815314e323dcc737dca890e09333246c02b2fc7c945487"} Sep 09 16:32:06.156755 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:06.156719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" event={"ID":"f78cdc1e-55e8-487b-bdb3-ca06eeefb5df","Type":"ContainerStarted","Data":"208f558da2e3f3c7b306153fdcef564079cb9350eb7a4de43ef817bfb46c70a5"} Sep 09 16:32:06.157166 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:06.156851 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:06.174814 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:06.174764 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" podStartSLOduration=1.465994603 podStartE2EDuration="2.174745383s" totalImagesPullingTime="708.75078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:06.172979275 +0000 UTC m=+2445.514171104" watchObservedRunningTime="2026-09-09 16:32:06.174745383 +0000 UTC m=+2445.515937212" Sep 09 16:32:10.923639 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:10.923601 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:32:12.167126 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.167094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8585bfb869-j668k" Sep 09 16:32:12.214736 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.214692 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:32:12.215298 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.214919 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" podUID="a5735c48-4c59-464d-af89-0f08c73bc1c4" containerName="maas-api" containerID="cri-o://a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb" gracePeriod=30 Sep 09 16:32:12.471911 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.471881 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:32:12.491437 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.491353 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs\") pod \"a5735c48-4c59-464d-af89-0f08c73bc1c4\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " Sep 09 16:32:12.491729 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.491459 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7stm\" (UniqueName: \"kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm\") pod \"a5735c48-4c59-464d-af89-0f08c73bc1c4\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " Sep 09 16:32:12.491729 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.491489 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca\") pod \"a5735c48-4c59-464d-af89-0f08c73bc1c4\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " Sep 09 16:32:12.491729 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.491526 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls\") pod \"a5735c48-4c59-464d-af89-0f08c73bc1c4\" (UID: \"a5735c48-4c59-464d-af89-0f08c73bc1c4\") " Sep 09 16:32:12.491995 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.491968 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5735c48-4c59-464d-af89-0f08c73bc1c4" (UID: "a5735c48-4c59-464d-af89-0f08c73bc1c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 16:32:12.494338 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.494172 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm" (OuterVolumeSpecName: "kube-api-access-x7stm") pod "a5735c48-4c59-464d-af89-0f08c73bc1c4" (UID: "a5735c48-4c59-464d-af89-0f08c73bc1c4"). InnerVolumeSpecName "kube-api-access-x7stm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 16:32:12.494338 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.494270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5735c48-4c59-464d-af89-0f08c73bc1c4" (UID: "a5735c48-4c59-464d-af89-0f08c73bc1c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:32:12.494338 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.494291 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5735c48-4c59-464d-af89-0f08c73bc1c4" (UID: "a5735c48-4c59-464d-af89-0f08c73bc1c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 16:32:12.593004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.592968 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7stm\" (UniqueName: \"kubernetes.io/projected/a5735c48-4c59-464d-af89-0f08c73bc1c4-kube-api-access-x7stm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:32:12.593004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.592997 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5735c48-4c59-464d-af89-0f08c73bc1c4-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:32:12.593004 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.593012 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:32:12.593239 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:12.593026 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5735c48-4c59-464d-af89-0f08c73bc1c4-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\"" Sep 09 16:32:13.198268 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.198235 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5735c48-4c59-464d-af89-0f08c73bc1c4" containerID="a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb" exitCode=0 Sep 09 16:32:13.198715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.198297 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" Sep 09 16:32:13.198715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.198359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" event={"ID":"a5735c48-4c59-464d-af89-0f08c73bc1c4","Type":"ContainerDied","Data":"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb"} Sep 09 16:32:13.198715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.198442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj" event={"ID":"a5735c48-4c59-464d-af89-0f08c73bc1c4","Type":"ContainerDied","Data":"bc8be494dd4a9b8053b0f4c16b57617aa54622edb14abf401f5275c15731e640"} Sep 09 16:32:13.198715 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.198492 2562 scope.go:117] "RemoveContainer" containerID="a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb" Sep 09 16:32:13.208341 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.208317 2562 scope.go:117] "RemoveContainer" containerID="a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb" Sep 09 16:32:13.208661 ip-10-0-130-57 kubenswrapper[2562]: E0909 16:32:13.208631 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb\": container with ID starting with a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb not found: ID does not exist" containerID="a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb" Sep 09 16:32:13.208780 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.208671 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb"} err="failed to get container status \"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb\": rpc error: code = NotFound desc = could not find container \"a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb\": container with ID starting with a89086c248d568d6351818996a5e54874e17db4969b45d8224e98915d165adeb not found: ID does not exist" Sep 09 16:32:13.215498 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.215473 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:32:13.218853 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:13.218825 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4b89c5b5-sf4mj"] Sep 09 16:32:15.190194 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:15.190151 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5735c48-4c59-464d-af89-0f08c73bc1c4" path="/var/lib/kubelet/pods/a5735c48-4c59-464d-af89-0f08c73bc1c4/volumes" Sep 09 16:32:16.760278 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:16.760232 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:32:21.738257 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:21.738224 2562 scope.go:117] "RemoveContainer" containerID="314116488d8af19cee8963e6f0e5a0bb92d74f2d425f888bbddc5afcb2c11505" Sep 09 16:32:23.830578 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:23.830523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8tlvm"] Sep 09 16:32:33.041752 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:33.041719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-scjp2_3cdc679a-6c26-406b-8223-49f2a4c44b06/manager/0.log" Sep 09 16:32:33.177156 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:33.177122 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zmlrh_fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5/manager/0.log" Sep 09 16:32:33.462841 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:33.462814 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w4nn8_8c401f4f-6576-4d25-aa7f-4e299da3de77/server/0.log" Sep 09 16:32:33.706574 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:33.706541 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-d5vn7_f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3/manager/0.log" Sep 09 16:32:33.717984 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:33.717924 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-d5vn7_f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3/copy-manifests/0.log" Sep 09 16:32:34.340119 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:34.340087 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8585bfb869-j668k_f78cdc1e-55e8-487b-bdb3-ca06eeefb5df/maas-api/0.log" Sep 09 16:32:35.743266 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:35.743223 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bfgph_6695ba1f-eae7-41cd-a52e-ca12927fe568/manager/0.log" Sep 09 16:32:35.862523 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:35.862491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9wwgv_cfdb095b-8755-4e8b-8ba1-5b63c7961fb5/manager/0.log" Sep 09 16:32:36.224992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:36.224963 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8tlvm_43b047e1-f0a3-48d0-be95-a509d962b640/limitador/0.log" Sep 09 16:32:36.716716 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:36.716677 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf_c2c70978-568b-4536-934d-632711d3636c/istio-proxy/0.log" Sep 09 16:32:37.213342 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:37.213306 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/1.log" Sep 09 16:32:38.453138 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:38.453108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs4ztk_c27aa575-0cdc-405e-bb53-ec7e71869820/main/0.log" Sep 09 16:32:38.683159 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:38.683129 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tc4pw_b150a0a3-f378-4f60-ae27-00cef5212f2c/main/0.log" Sep 09 16:32:47.154685 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:47.154649 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t5z2s_04238531-1ed9-461f-ac8d-d46b0af52bb2/global-pull-secret-syncer/0.log" Sep 09 16:32:47.282926 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:47.282891 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rqfdc_881e116f-eb06-4df6-87a9-2cdf700fdd72/konnectivity-agent/0.log" Sep 09 16:32:47.304542 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:47.304512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-57.ec2.internal_b59b194d8dabcdf6b93ea8511f43438d/haproxy/0.log" Sep 09 16:32:51.581885 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:51.581855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bfgph_6695ba1f-eae7-41cd-a52e-ca12927fe568/manager/0.log" Sep 09 16:32:51.611355 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:51.611308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-9wwgv_cfdb095b-8755-4e8b-8ba1-5b63c7961fb5/manager/0.log" Sep 09 16:32:51.743464 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:51.743399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8tlvm_43b047e1-f0a3-48d0-be95-a509d962b640/limitador/0.log" Sep 09 16:32:53.408291 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.408256 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/alertmanager/0.log" Sep 09 16:32:53.432573 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.432542 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/config-reloader/0.log" Sep 09 16:32:53.456484 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.456450 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/kube-rbac-proxy-web/0.log" Sep 09 16:32:53.478463 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.478434 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/kube-rbac-proxy/0.log" Sep 09 16:32:53.498881 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.498851 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/kube-rbac-proxy-metric/0.log" Sep 09 16:32:53.519308 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.519277 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/prom-label-proxy/0.log" Sep 09 16:32:53.539110 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.539085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9069f88-8c0c-46bf-8d0c-cfa299f54513/init-config-reloader/0.log" Sep 09 16:32:53.574270 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.574194 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-k6ppp_3677b106-49bd-4ac2-8f4e-0e7d2bd4dea7/cluster-monitoring-operator/0.log" Sep 09 16:32:53.732522 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.732492 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qk4f_66001ff9-94de-4765-88f4-450c64c01efc/node-exporter/0.log" Sep 09 16:32:53.758580 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.758541 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qk4f_66001ff9-94de-4765-88f4-450c64c01efc/kube-rbac-proxy/0.log" Sep 09 16:32:53.780257 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:53.780230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qk4f_66001ff9-94de-4765-88f4-450c64c01efc/init-textfile/0.log" Sep 09 16:32:54.058667 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.058638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/prometheus/0.log" Sep 09 16:32:54.079524 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.079477 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/config-reloader/0.log" Sep 09 16:32:54.102037 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.102004 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/thanos-sidecar/0.log" Sep 09 16:32:54.124304 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.124229 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/kube-rbac-proxy-web/0.log" Sep 09 16:32:54.144722 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.144689 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/kube-rbac-proxy/0.log" Sep 09 16:32:54.165377 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.165357 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/kube-rbac-proxy-thanos/0.log" Sep 09 16:32:54.187752 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.187724 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_27333cea-cd69-46ed-9261-1e9577f8c46a/init-config-reloader/0.log" Sep 09 16:32:54.268310 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.268289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-lncpf_d4ed807b-b4ef-445d-82b3-01003ac5a1bb/prometheus-operator-admission-webhook/0.log" Sep 09 16:32:54.303609 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.303572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b99c5c8f9-bp9wz_d194cb63-9e92-49de-b17a-c3ce77797d16/telemeter-client/0.log" Sep 09 16:32:54.328709 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.328678 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b99c5c8f9-bp9wz_d194cb63-9e92-49de-b17a-c3ce77797d16/reload/0.log" Sep 09 16:32:54.354100 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.354070 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b99c5c8f9-bp9wz_d194cb63-9e92-49de-b17a-c3ce77797d16/kube-rbac-proxy/0.log" Sep 09 16:32:54.392899 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.392826 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/thanos-query/0.log" Sep 09 16:32:54.415682 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.415644 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/kube-rbac-proxy-web/0.log" Sep 09 16:32:54.444811 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.444778 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/kube-rbac-proxy/0.log" Sep 09 16:32:54.464471 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.464448 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/prom-label-proxy/0.log" Sep 09 16:32:54.486331 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.486305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/kube-rbac-proxy-rules/0.log" Sep 09 16:32:54.507721 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:54.507690 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b97b9786-k62bs_215b1af2-8686-45db-a869-efb1ab5009b3/kube-rbac-proxy-metrics/0.log" Sep 09 16:32:55.997265 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:55.997230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-d7kfz_c40bf4af-9876-408d-9527-5115d436f397/networking-console-plugin/0.log" Sep 09 16:32:56.613249 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:56.613214 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/1.log" Sep 09 16:32:56.618373 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:56.618348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-rl7db_d26dd5f9-339d-4123-b5e7-bf490e802699/console-operator/2.log" Sep 09 16:32:57.157420 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:57.157391 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-rmfsv_e326c70e-e584-422a-b300-a8bcabd58ebb/download-server/0.log" Sep 09 16:32:57.812566 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:57.812536 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-rjthl_7c3aaeaa-985d-4ea4-a623-2522319a941a/volume-data-source-validator/0.log" Sep 09 16:32:59.027882 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:59.027842 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wbgbv_ef9d60a4-51da-4d45-b1fa-64a1d251965f/dns/0.log" Sep 09 16:32:59.047876 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:59.047854 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wbgbv_ef9d60a4-51da-4d45-b1fa-64a1d251965f/kube-rbac-proxy/0.log" Sep 09 16:32:59.122449 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:59.122399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nt5bs_b783ea01-682c-4b85-adea-0c3cff2e3734/dns-node-resolver/0.log" Sep 09 16:32:59.698466 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:32:59.698428 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6m4g_65c46cff-df1c-4afe-9ee8-a8e48413f0c1/node-ca/0.log" Sep 09 16:33:00.569177 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:00.569147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf_c2c70978-568b-4536-934d-632711d3636c/istio-proxy/0.log" Sep 09 16:33:00.869475 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:00.869446 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/0.log" Sep 09 16:33:00.877949 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:00.877926 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/1.log" Sep 09 16:33:01.886147 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:01.886121 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srs75_9bcb88f1-3276-4896-bbe6-e6b29f66d4a1/serve-healthcheck-canary/0.log" Sep 09 16:33:02.374018 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.373985 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bnh8_39132dab-f7cc-40b4-8f79-d4dd024cc581/kube-rbac-proxy/0.log" Sep 09 16:33:02.394234 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.394209 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bnh8_39132dab-f7cc-40b4-8f79-d4dd024cc581/exporter/0.log" Sep 09 16:33:02.414872 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.414850 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2bnh8_39132dab-f7cc-40b4-8f79-d4dd024cc581/extractor/0.log" Sep 09 16:33:02.986562 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.986527 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr"] Sep 09 16:33:02.987103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.987071 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5735c48-4c59-464d-af89-0f08c73bc1c4" containerName="maas-api" Sep 09 16:33:02.987103 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.987091 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5735c48-4c59-464d-af89-0f08c73bc1c4" containerName="maas-api" Sep 09 16:33:02.987282 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.987172 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5735c48-4c59-464d-af89-0f08c73bc1c4" containerName="maas-api" Sep 09 16:33:02.990351 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.990335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:02.992830 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.992807 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6q5c\"/\"kube-root-ca.crt\"" Sep 09 16:33:02.994198 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.994173 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6q5c\"/\"openshift-service-ca.crt\"" Sep 09 16:33:02.994299 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:02.994210 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g6q5c\"/\"default-dockercfg-92n2g\"" Sep 09 16:33:03.000346 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.000323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr"] Sep 09 16:33:03.086233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.086204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-podres\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.086233 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.086235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-proc\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.086506 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.086293 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85pn\" (UniqueName: \"kubernetes.io/projected/37946ecd-789d-417b-bce9-d8cb024fe777-kube-api-access-t85pn\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.086506 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.086320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-lib-modules\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.086506 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.086339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-sys\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.186891 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.186860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-podres\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.186896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-proc\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.186968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85pn\" (UniqueName: \"kubernetes.io/projected/37946ecd-789d-417b-bce9-d8cb024fe777-kube-api-access-t85pn\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-lib-modules\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187057 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-sys\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-proc\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-podres\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-sys\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.187251 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.187126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37946ecd-789d-417b-bce9-d8cb024fe777-lib-modules\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.201992 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.201961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85pn\" (UniqueName: \"kubernetes.io/projected/37946ecd-789d-417b-bce9-d8cb024fe777-kube-api-access-t85pn\") pod \"perf-node-gather-daemonset-lxpkr\" (UID: \"37946ecd-789d-417b-bce9-d8cb024fe777\") " pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.300717 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.300626 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:03.573238 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:03.573211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr"] Sep 09 16:33:03.575453 ip-10-0-130-57 kubenswrapper[2562]: W0909 16:33:03.575386 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37946ecd_789d_417b_bce9_d8cb024fe777.slice/crio-7c941e2d82e72fda37a3ace33360bc03a5b2b27289777a30395a87a36c0096f2 WatchSource:0}: Error finding container 7c941e2d82e72fda37a3ace33360bc03a5b2b27289777a30395a87a36c0096f2: Status 404 returned error can't find the container with id 7c941e2d82e72fda37a3ace33360bc03a5b2b27289777a30395a87a36c0096f2 Sep 09 16:33:04.460672 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:04.460634 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" event={"ID":"37946ecd-789d-417b-bce9-d8cb024fe777","Type":"ContainerStarted","Data":"6e836bec711c2f0a3093e088f1f8b7c2f8788236123adab3e9f60fdf83f19688"} Sep 09 16:33:04.461054 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:04.460694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" event={"ID":"37946ecd-789d-417b-bce9-d8cb024fe777","Type":"ContainerStarted","Data":"7c941e2d82e72fda37a3ace33360bc03a5b2b27289777a30395a87a36c0096f2"} Sep 09 16:33:04.461054 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:04.460777 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:04.477228 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:04.477180 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" podStartSLOduration=2.477165483 podStartE2EDuration="2.477165483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:04.474820388 +0000 UTC m=+2503.816012222" watchObservedRunningTime="2026-09-09 16:33:04.477165483 +0000 UTC m=+2503.818357312" Sep 09 16:33:04.828904 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:04.828816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-scjp2_3cdc679a-6c26-406b-8223-49f2a4c44b06/manager/0.log" Sep 09 16:33:05.000751 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:05.000717 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zmlrh_fa7885a2-c0c8-46f5-88c5-911bf0f9e9c5/manager/0.log" Sep 09 16:33:05.063840 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:05.063814 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w4nn8_8c401f4f-6576-4d25-aa7f-4e299da3de77/server/0.log" Sep 09 16:33:05.210392 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:05.210360 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-d5vn7_f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3/manager/0.log" Sep 09 16:33:05.226703 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:05.226682 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4ccd886c-d5vn7_f9ccdb23-9f94-4c75-9d91-5be4e1ac66b3/copy-manifests/0.log" Sep 09 16:33:06.497059 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:06.497027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-678f94cf5f-h886r_2b7067f9-9854-4b29-81cf-d6f41c4da5d4/manager/0.log" Sep 09 16:33:07.438969 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:07.438937 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf_c2c70978-568b-4536-934d-632711d3636c/istio-proxy/0.log" Sep 09 16:33:07.697786 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:07.697686 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/0.log" Sep 09 16:33:07.705993 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:07.705965 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/1.log" Sep 09 16:33:09.210624 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:09.210595 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q84gf_c2c70978-568b-4536-934d-632711d3636c/istio-proxy/0.log" Sep 09 16:33:09.481850 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:09.481682 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/0.log" Sep 09 16:33:09.490675 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:09.490648 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vrq8h_13b41dac-ff54-4a7a-bd94-0dc9e85bf1d3/istio-proxy/1.log" Sep 09 16:33:10.476365 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:10.476328 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g6q5c/perf-node-gather-daemonset-lxpkr" Sep 09 16:33:11.392764 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:11.392735 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-48wck_788fb6c6-44f6-469e-86e4-6bd076616049/migrator/0.log" Sep 09 16:33:11.413501 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:11.413467 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-48wck_788fb6c6-44f6-469e-86e4-6bd076616049/graceful-termination/0.log" Sep 09 16:33:13.106942 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.106910 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/kube-multus-additional-cni-plugins/0.log" Sep 09 16:33:13.132442 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.132374 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/egress-router-binary-copy/0.log" Sep 09 16:33:13.157905 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.157882 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/cni-plugins/0.log" Sep 09 16:33:13.182049 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.182025 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/bond-cni-plugin/0.log" Sep 09 16:33:13.207003 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.206976 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/routeoverride-cni/0.log" Sep 09 16:33:13.229548 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.229524 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/whereabouts-cni-bincopy/0.log" Sep 09 16:33:13.253551 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.253483 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2vnvh_82d45f58-1805-4c8d-978e-09aab955a0a8/whereabouts-cni/0.log" Sep 09 16:33:13.653284 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.653252 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q2vlw_16db5106-107a-491f-95f0-0b2012d6deeb/kube-multus/0.log" Sep 09 16:33:13.774951 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.774921 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wpwhx_337bfd9d-63ad-43b9-9b6f-e79aab96a48f/network-metrics-daemon/0.log" Sep 09 16:33:13.794281 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:13.794249 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wpwhx_337bfd9d-63ad-43b9-9b6f-e79aab96a48f/kube-rbac-proxy/0.log" Sep 09 16:33:15.141788 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.141759 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/ovn-controller/0.log" Sep 09 16:33:15.159427 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.159388 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/ovn-acl-logging/0.log" Sep 09 16:33:15.184594 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.184563 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/kube-rbac-proxy-node/0.log" Sep 09 16:33:15.204186 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.204165 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 16:33:15.221091 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.221056 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/northd/0.log" Sep 09 16:33:15.240048 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.240030 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/nbdb/0.log" Sep 09 16:33:15.260893 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.260876 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/sbdb/0.log" Sep 09 16:33:15.329893 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:15.329865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfn9r_4dc49812-c7e5-4525-85ed-01be2448c4d0/ovnkube-controller/0.log" Sep 09 16:33:16.394311 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:16.394281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-gt8ms_1eeafd31-c7ce-4911-98bd-773a62b87ba5/check-endpoints/0.log" Sep 09 16:33:16.417025 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:16.416994 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8kgr4_dd51558a-dfb7-4a0a-b9b6-0b69958d841e/network-check-target-container/0.log" Sep 09 16:33:17.546391 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:17.546362 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5h7ll_4b746718-910e-48d2-b23c-4f25fab41d05/iptables-alerter/0.log" Sep 09 16:33:18.252659 ip-10-0-130-57 kubenswrapper[2562]: I0909 16:33:18.252618 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4c4rv_7f6ea669-ea32-4fbb-8337-85e2e00a0a56/tuned/0.log"