Sep 20 01:52:26.876333 ip-10-0-137-50 systemd[1]: Starting Kubernetes Kubelet... Sep 20 01:52:27.307051 ip-10-0-137-50 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 20 01:52:27.307051 ip-10-0-137-50 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 20 01:52:27.307051 ip-10-0-137-50 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 20 01:52:27.307672 ip-10-0-137-50 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 20 01:52:27.307672 ip-10-0-137-50 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 20 01:52:27.309456 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.309373 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 20 01:52:27.312576 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312561 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.312576 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312576 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312580 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312583 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312586 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312589 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312592 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312595 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312597 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312600 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312603 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312605 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312608 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312610 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312612 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312615 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312618 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312620 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312622 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312625 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.312636 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312643 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312646 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312649 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312651 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312653 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312656 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312658 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312661 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312663 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312665 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312668 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312670 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312672 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312675 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312677 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312680 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312683 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312685 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312687 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312690 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.313082 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312692 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312695 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312697 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312700 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312704 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312707 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312709 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312712 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312714 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312717 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312719 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312721 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312723 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312726 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312729 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312731 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312734 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312740 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312742 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.313547 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312745 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312747 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312750 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312752 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312755 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312757 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312760 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312762 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312765 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312767 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312770 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312772 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312776 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312779 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312782 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312785 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312789 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312792 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312796 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.314048 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312798 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312801 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312803 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312806 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312808 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312810 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312813 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312815 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312818 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312821 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312823 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312826 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312829 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312832 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312834 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312836 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312839 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312841 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312844 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312846 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.314487 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.312849 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313517 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313524 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313527 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313530 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313533 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313536 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313539 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313542 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313545 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313548 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313551 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313554 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313557 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313560 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313562 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313565 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313567 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313570 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313573 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.314962 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313576 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313578 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313581 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313584 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313586 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313589 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313591 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313595 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313598 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313601 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313603 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313606 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313609 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313611 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313614 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313616 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313619 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313621 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313623 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.315462 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313626 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313628 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313631 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313634 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313636 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313638 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313641 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313643 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313645 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313647 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313650 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313652 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313656 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313658 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313662 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313664 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313667 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313669 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313672 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.315913 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313674 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313677 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313679 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313681 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313684 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313686 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313689 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313691 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313695 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313698 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313700 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313702 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313705 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313708 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313710 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313712 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313719 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313721 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313724 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313726 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.316376 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313728 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313731 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313733 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313735 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313738 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313740 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313744 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313746 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313749 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313751 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313753 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313756 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313759 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313770 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313774 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313776 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313779 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313782 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313784 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313787 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.316831 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313789 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.313791 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314166 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314171 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314174 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314176 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314179 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314181 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314184 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314187 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314189 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314192 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314244 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314248 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314251 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314253 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314256 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314259 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314262 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314266 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.317321 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314269 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314272 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314275 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314278 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314280 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314283 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314287 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314291 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314294 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314296 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314298 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314302 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314304 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314307 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314309 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314311 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314314 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314316 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314318 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.317793 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314321 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314323 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314326 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314329 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314331 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314334 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314336 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314338 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314341 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314343 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314348 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314351 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314353 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314356 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314358 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314360 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314363 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314365 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314368 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314370 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.318269 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314372 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314375 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314377 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314379 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314382 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314384 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314386 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314391 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314394 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314397 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314399 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314402 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314405 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314407 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314410 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314413 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314415 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314418 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314421 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314423 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.318771 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314425 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314428 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314431 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314434 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314437 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314439 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314441 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314444 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314446 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314449 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314452 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314454 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314456 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314459 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314461 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314463 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314466 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314468 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314471 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.319248 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314473 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314476 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.314478 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314556 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314596 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314604 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314608 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314612 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314615 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314620 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314624 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314627 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314630 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314633 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314637 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314642 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314645 2578 flags.go:64] FLAG: --cgroup-root="" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314648 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314651 2578 flags.go:64] FLAG: --client-ca-file="" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314654 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314657 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314661 2578 flags.go:64] FLAG: --cluster-domain="" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314663 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314666 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 20 01:52:27.319694 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314669 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314672 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314676 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314679 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314683 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314686 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314688 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314691 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314694 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314697 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314700 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314704 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314707 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314709 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314712 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314715 2578 flags.go:64] FLAG: --enable-server="true" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314717 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314722 2578 flags.go:64] FLAG: --event-burst="100" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314725 2578 flags.go:64] FLAG: --event-qps="50" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314728 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314731 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314734 2578 flags.go:64] FLAG: --eviction-hard="" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314737 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314741 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314745 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 20 01:52:27.320270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314748 2578 flags.go:64] FLAG: --eviction-soft="" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314750 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314753 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314756 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314759 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314761 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314764 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314767 2578 flags.go:64] FLAG: --feature-gates="" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314770 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314773 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314776 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314779 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314782 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314784 2578 flags.go:64] FLAG: --help="false" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314787 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314790 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314792 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314795 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314798 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314801 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314804 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314807 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314810 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 20 01:52:27.320836 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314812 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314815 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314818 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314821 2578 flags.go:64] FLAG: --kube-reserved="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314824 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314827 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314829 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314833 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314839 2578 flags.go:64] FLAG: --lock-file="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314841 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314844 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314847 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314852 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314855 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314857 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314860 2578 flags.go:64] FLAG: --logging-format="text" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314863 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314866 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314868 2578 flags.go:64] FLAG: --manifest-url="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314871 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314875 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314878 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314882 2578 flags.go:64] FLAG: --max-pods="110" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314885 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314888 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 20 01:52:27.321406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314891 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314893 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314896 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314899 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314901 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314908 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314911 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314914 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314916 2578 flags.go:64] FLAG: --pod-cidr="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314919 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314924 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314927 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314930 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314933 2578 flags.go:64] FLAG: --port="10250" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314936 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314939 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-010664bec20b87c01" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314943 2578 flags.go:64] FLAG: --qos-reserved="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314946 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314949 2578 flags.go:64] FLAG: --register-node="true" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314952 2578 flags.go:64] FLAG: --register-with-taints="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314955 2578 flags.go:64] FLAG: --registry-burst="10" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314958 2578 flags.go:64] FLAG: --registry-qps="5" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314961 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314963 2578 flags.go:64] FLAG: --reserved-memory="" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314967 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 20 01:52:27.322042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314970 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314973 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314976 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314979 2578 flags.go:64] FLAG: --runonce="false" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314981 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314984 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314987 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314990 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314993 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314995 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.314998 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315001 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315003 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315006 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315009 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315011 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315014 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315017 2578 flags.go:64] FLAG: --system-cgroups="" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315019 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315038 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315041 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315045 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315049 2578 flags.go:64] FLAG: --tls-min-version="" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315052 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315055 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 20 01:52:27.322644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315058 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315061 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315064 2578 flags.go:64] FLAG: --v="2" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315067 2578 flags.go:64] FLAG: --version="false" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315071 2578 flags.go:64] FLAG: --vmodule="" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315074 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315077 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315194 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315199 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315202 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315206 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315209 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315212 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315215 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315217 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315220 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315222 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315225 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315227 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315230 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315232 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.323236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315234 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315237 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315239 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315242 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315244 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315247 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315249 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315254 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315257 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315260 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315263 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315265 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315268 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315270 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315273 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315275 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315277 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315280 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315282 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.323741 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315286 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315289 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315291 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315294 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315296 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315298 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315301 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315304 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315306 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315309 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315311 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315313 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315316 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315318 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315320 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315323 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315325 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315328 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315330 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315332 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.324217 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315336 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315339 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315342 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315345 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315347 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315350 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315352 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315355 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315357 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315360 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315362 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315364 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315367 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315369 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315372 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315374 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315376 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315379 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315381 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315384 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.324688 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315386 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315388 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315391 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315393 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315395 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315398 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315400 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315402 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315405 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315407 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315409 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315412 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315415 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315418 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315421 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315424 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315426 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315429 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315431 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315433 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.325216 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315436 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315438 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315441 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315443 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315446 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.315448 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.325703 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.315453 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 20 01:52:27.329318 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.329295 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 20 01:52:27.329318 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.329316 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.329341 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.329353 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329431 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329439 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329444 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329449 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329454 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329458 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329462 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.329463 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329467 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329471 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329475 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329479 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329483 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329487 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329491 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329495 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329499 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329503 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329507 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329511 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329515 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329520 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329524 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329527 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329531 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329536 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329540 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329543 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.329893 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329548 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329554 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329560 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329566 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329570 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329574 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329579 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329583 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329587 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329591 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329595 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329598 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329605 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329611 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329616 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329620 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329625 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329629 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329633 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.330718 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329637 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329641 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329645 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329649 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329653 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329658 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329662 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329666 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329670 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329674 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329678 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329683 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329687 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329691 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329694 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329699 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329708 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329712 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329716 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329720 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.331241 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329724 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329728 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329732 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329736 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329739 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329743 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329748 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329753 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329757 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329771 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329778 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329782 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329786 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329790 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329794 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329798 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329802 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329806 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329810 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.331944 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329814 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329818 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329822 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329826 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329830 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329834 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329838 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329842 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329846 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329850 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329855 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329859 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329863 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.329866 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.332708 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.329874 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330072 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330082 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330087 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330091 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330096 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330100 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330105 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330110 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330114 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330118 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330122 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330126 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330131 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330135 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330139 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330144 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330150 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330154 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 20 01:52:27.333529 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330158 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330162 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330166 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330170 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330174 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330178 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330182 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330186 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330190 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330194 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330199 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330203 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330208 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330211 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330215 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330219 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330223 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330227 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330232 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330236 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 20 01:52:27.334010 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330241 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330245 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330249 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330255 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330261 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330265 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330270 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330274 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330279 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330283 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330287 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330291 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330295 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330299 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330303 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330307 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330310 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330314 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330318 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330322 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 20 01:52:27.334509 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330326 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330330 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330334 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330339 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330344 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330348 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330352 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330356 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330360 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330364 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330368 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330372 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330376 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330380 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330384 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330388 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330392 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330396 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330400 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 20 01:52:27.334975 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330404 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330408 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330412 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330416 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330420 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330424 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330427 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330431 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330435 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330439 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330443 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330447 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330451 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330455 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330459 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330463 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330467 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330471 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330475 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330479 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 20 01:52:27.335437 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330483 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:27.330487 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.330495 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.330659 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.334754 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.335821 2578 server.go:1019] "Starting client certificate rotation" Sep 20 01:52:27.335931 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.335915 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 20 01:52:27.336105 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.335951 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 20 01:52:27.338656 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.338642 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 20 01:52:27.339756 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.339739 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 20 01:52:27.357069 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.357050 2578 log.go:25] "Validated CRI v1 runtime API" Sep 20 01:52:27.358771 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.358749 2578 log.go:25] "Validated CRI v1 image API" Sep 20 01:52:27.359508 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.359485 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 20 01:52:27.362048 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.362010 2578 fs.go:135] Filesystem UUIDs: map[2566f699-f289-42a6-9784-c4a607011f88:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dd496075-fbe1-4cc2-8afd-d589ec4706e7:/dev/nvme0n1p3] Sep 20 01:52:27.362110 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.362047 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 20 01:52:27.367788 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.367769 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 20 01:52:27.368153 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.368048 2578 manager.go:216] Machine: {Timestamp:2026-09-20 01:52:27.366629928 +0000 UTC m=+0.379827446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3209795 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c73ced94c2fa8c2d077bad54d8c77 SystemUUID:ec2c73ce-d94c-2fa8-c2d0-77bad54d8c77 BootID:1c2f31b4-f44d-404e-8b74-360d09345c47 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:e2:5b:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:e2:5b:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:e1:e4:c5:b8:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 20 01:52:27.368153 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.368146 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 20 01:52:27.368272 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.368229 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 20 01:52:27.369945 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.369915 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 20 01:52:27.370097 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.369949 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 20 01:52:27.370860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.370850 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 20 01:52:27.370896 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.370863 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 20 01:52:27.370896 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.370870 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 20 01:52:27.371644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.371634 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 20 01:52:27.371718 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.371710 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 20 01:52:27.376033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376011 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 20 01:52:27.376143 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376134 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 20 01:52:27.376181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376166 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 20 01:52:27.376181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376176 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 20 01:52:27.376234 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376192 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 20 01:52:27.376234 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376207 2578 kubelet.go:398] "Adding apiserver pod source" Sep 20 01:52:27.376234 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376216 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 20 01:52:27.376645 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376624 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 20 01:52:27.376703 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.376659 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 20 01:52:27.382384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.382363 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 20 01:52:27.385432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.385410 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 20 01:52:27.385531 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.385449 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 20 01:52:27.385626 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.385610 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxjp6" Sep 20 01:52:27.386947 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386927 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386953 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386961 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386968 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386979 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.386987 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.386986 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387006 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 20 01:52:27.387014 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387015 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387040 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.387040 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387049 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387080 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387087 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387108 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 20 01:52:27.387284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.387114 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 20 01:52:27.390869 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.390855 2578 server.go:1268] "Started kubelet" Sep 20 01:52:27.390957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.390926 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 20 01:52:27.391006 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.390948 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 20 01:52:27.391082 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.391018 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 20 01:52:27.391780 ip-10-0-137-50 systemd[1]: Started Kubernetes Kubelet. Sep 20 01:52:27.393219 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.393185 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxjp6" Sep 20 01:52:27.393632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.393617 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 20 01:52:27.395065 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.395048 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 20 01:52:27.395413 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.395397 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 20 01:52:27.396568 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.396553 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 20 01:52:27.397657 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.396592 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-50.ec2.internal.18d6e481a4a953c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-50.ec2.internal,UID:ip-10-0-137-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-50.ec2.internal,},FirstTimestamp:2026-09-20 01:52:27.390825414 +0000 UTC m=+0.404022932,LastTimestamp:2026-09-20 01:52:27.390825414 +0000 UTC m=+0.404022932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-50.ec2.internal,}" Sep 20 01:52:27.404235 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.404221 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 20 01:52:27.404309 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.404292 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 20 01:52:27.404411 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.404394 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.404984 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.404965 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 20 01:52:27.404984 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.404983 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 20 01:52:27.405140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.404994 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 20 01:52:27.405140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405090 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 20 01:52:27.405140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405099 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 20 01:52:27.405273 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405194 2578 factory.go:55] Registering systemd factory Sep 20 01:52:27.405273 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405207 2578 factory.go:223] Registration of the systemd container factory successfully Sep 20 01:52:27.405587 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405497 2578 factory.go:153] Registering CRI-O factory Sep 20 01:52:27.405587 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405512 2578 factory.go:223] Registration of the crio container factory successfully Sep 20 01:52:27.405587 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405574 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 20 01:52:27.405730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405593 2578 factory.go:103] Registering Raw factory Sep 20 01:52:27.405730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405603 2578 manager.go:1196] Started watching for new ooms in manager Sep 20 01:52:27.405969 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.405951 2578 manager.go:349] Starting recovery of all containers Sep 20 01:52:27.407721 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.407691 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 20 01:52:27.407836 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.407794 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 20 01:52:27.417910 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.417894 2578 manager.go:354] Recovery completed Sep 20 01:52:27.421923 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.421910 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.423562 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.423547 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.423633 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.423577 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.423633 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.423588 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.424172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424159 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 20 01:52:27.424225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424173 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 20 01:52:27.424225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424191 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 20 01:52:27.424225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424160 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.424841 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424828 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.424887 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424851 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.424887 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.424860 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.426310 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.426299 2578 policy_none.go:49] "None policy: Start" Sep 20 01:52:27.426347 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.426315 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 20 01:52:27.426347 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.426324 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 20 01:52:27.428201 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.428190 2578 policy_none.go:47] "Start" Sep 20 01:52:27.468766 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.468596 2578 manager.go:337] "Starting Device Plugin manager" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.468793 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.468806 2578 server.go:87] "Starting device plugin registration server" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.469174 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.469196 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.469282 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.469362 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.469369 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.469969 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 20 01:52:27.489401 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.470005 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.512543 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.512504 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 20 01:52:27.513812 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.513785 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 20 01:52:27.513920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.513835 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 20 01:52:27.513920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.513860 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 20 01:52:27.514008 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.513941 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 20 01:52:27.516456 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.516435 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:27.569713 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.569623 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.570710 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.570695 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.570820 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.570722 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.570820 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.570734 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.570820 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.570761 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.577037 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.577009 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.577083 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.577046 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-50.ec2.internal\": node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.590674 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.590657 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.614144 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.614123 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal"] Sep 20 01:52:27.614198 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.614190 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.615696 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.615675 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.615744 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.615703 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.615744 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.615715 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.617992 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.617980 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.618138 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.618194 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618158 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.618721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618706 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.618781 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618733 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.618781 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618749 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.618841 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618710 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.618841 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618807 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.618841 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.618821 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.621085 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.621072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.621157 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.621095 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 20 01:52:27.621810 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.621784 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientMemory" Sep 20 01:52:27.621904 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.621819 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasNoDiskPressure" Sep 20 01:52:27.621904 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.621830 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeHasSufficientPID" Sep 20 01:52:27.649647 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.649621 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-50.ec2.internal\" not found" node="ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.653812 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.653793 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-50.ec2.internal\" not found" node="ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.691055 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.691010 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.705974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.705943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.706101 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.705988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.706101 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.706053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c472519add22672739b1e4cfa2793bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-50.ec2.internal\" (UID: \"c472519add22672739b1e4cfa2793bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.791507 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.791471 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.806873 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c472519add22672739b1e4cfa2793bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-50.ec2.internal\" (UID: \"c472519add22672739b1e4cfa2793bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.806974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.806974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.806974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c472519add22672739b1e4cfa2793bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-50.ec2.internal\" (UID: \"c472519add22672739b1e4cfa2793bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.806974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.806974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.806947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1391da81b03a052e00ea4b68a886b3f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal\" (UID: \"e1391da81b03a052e00ea4b68a886b3f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.892336 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.892266 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:27.952749 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.952723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.956875 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:27.956854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:27.992953 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:27.992915 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:28.093472 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.093435 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:28.194126 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.194009 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-50.ec2.internal\" not found" Sep 20 01:52:28.264471 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.264442 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:28.304485 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.304451 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" Sep 20 01:52:28.315231 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.315213 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 20 01:52:28.316990 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.316976 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" Sep 20 01:52:28.322973 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.322956 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 20 01:52:28.336084 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.336064 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 20 01:52:28.336191 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.336174 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 20 01:52:28.336238 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.336221 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 20 01:52:28.377380 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.377350 2578 apiserver.go:52] "Watching apiserver" Sep 20 01:52:28.393893 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.393874 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 20 01:52:28.395761 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.395733 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf","openshift-image-registry/node-ca-dczpm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal","openshift-multus/multus-additional-cni-plugins-89dck","openshift-multus/multus-tz2wh","openshift-multus/network-metrics-daemon-xwfcm","openshift-ovn-kubernetes/ovnkube-node-7b7rr","kube-system/global-pull-secret-syncer-5j9ht","kube-system/konnectivity-agent-v24kh","kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dd47k","openshift-dns/node-resolver-xpl5v","openshift-network-diagnostics/network-check-target-pp4rc","openshift-network-operator/iptables-alerter-pkjw4"] Sep 20 01:52:28.398421 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.398401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.400598 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.400577 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.400702 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.400604 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-762g8\"" Sep 20 01:52:28.400702 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.400617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.400702 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.400607 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.400702 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.400655 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 20 01:52:28.401380 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.401334 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-19 01:47:28 +0000 UTC" deadline="2028-02-18 01:35:42.263494517 +0000 UTC" Sep 20 01:52:28.401435 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.401382 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12383h43m13.862116027s" Sep 20 01:52:28.402691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.402675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-khpdd\"" Sep 20 01:52:28.402691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.402687 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.402987 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.402967 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.402987 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.402981 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 20 01:52:28.403166 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.403091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.404527 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.404510 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 20 01:52:28.404963 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.404941 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 20 01:52:28.405129 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405107 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 20 01:52:28.405302 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.405379 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405310 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5r49k\"" Sep 20 01:52:28.405379 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405333 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 20 01:52:28.405567 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405505 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.405567 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.405563 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.407193 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.407175 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 20 01:52:28.407264 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.407204 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mtz2t\"" Sep 20 01:52:28.407541 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.407522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.407620 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.407601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:28.409860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-bin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.409960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.409960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-etc-kubernetes\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.409960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.409960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.409960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-multus-certs\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.409987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-multus\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-kubelet\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-daemon-config\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-socket-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-registration-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89ctn\" (UniqueName: \"kubernetes.io/projected/0dd72a65-aeeb-4cef-844d-0277bfde3433-kube-api-access-89ctn\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-binary-copy\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-netns\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-hostroot\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cnibin\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-k8s-cni-cncf-io\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsdl\" (UniqueName: \"kubernetes.io/projected/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-kube-api-access-cnsdl\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-system-cni-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmcgf\" (UniqueName: \"kubernetes.io/projected/e72130d2-cf4b-4cb7-9912-a7fe11007acd-kube-api-access-vmcgf\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-system-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-conf-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-device-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw7z\" (UniqueName: \"kubernetes.io/projected/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-kube-api-access-fbw7z\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cnibin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-sys-fs\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-host\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-serviceca\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410811 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cni-binary-copy\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.410920 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.410825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-socket-dir-parent\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.412216 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.412307 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.412294 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:28.412385 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412367 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 20 01:52:28.412547 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412525 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.412873 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412842 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.412966 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 20 01:52:28.412966 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412960 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8kn9\"" Sep 20 01:52:28.413090 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.412900 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 20 01:52:28.413090 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.413004 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 20 01:52:28.414504 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.414469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.416548 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.416528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 20 01:52:28.416640 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.416529 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 20 01:52:28.416640 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.416531 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r4nnl\"" Sep 20 01:52:28.417393 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.417377 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.417494 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.417476 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 20 01:52:28.419091 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.419060 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8g865\"" Sep 20 01:52:28.419179 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.419153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.419222 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.419204 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.419570 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.419558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.421462 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.421445 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.421556 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.421464 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sjbdm\"" Sep 20 01:52:28.421556 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.421469 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.421987 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.421970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:28.422080 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.422046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:28.424366 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.424347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.426227 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.426209 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 20 01:52:28.426404 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.426374 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c4f4k\"" Sep 20 01:52:28.426481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.426405 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 20 01:52:28.427203 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.426753 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 20 01:52:28.436981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.436963 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwtpp" Sep 20 01:52:28.444750 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.444698 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwtpp" Sep 20 01:52:28.506609 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.506587 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 20 01:52:28.511558 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cnibin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.511650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.511650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-modprobe-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.511650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-conf\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.511650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-host\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.511650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-node-log\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511648 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cnibin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-script-lib\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511684 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-host\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysconfig\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8q2f\" (UniqueName: \"kubernetes.io/projected/2c5c4960-f167-4d58-999e-c5945302ede8-kube-api-access-b8q2f\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-serviceca\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.511826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cni-binary-copy\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-socket-dir-parent\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-host\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-socket-dir-parent\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-lib-modules\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.511981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da7c700-b7cc-4c87-9400-ef17a3557825-hosts-file\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-kubelet\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-systemd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512107 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512142 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-serviceca\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.512172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da7c700-b7cc-4c87-9400-ef17a3557825-tmp-dir\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-registration-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-binary-copy\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-registration-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mk74\" (UniqueName: \"kubernetes.io/projected/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-kube-api-access-7mk74\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-bin\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512267 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-cni-binary-copy\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrngq\" (UniqueName: \"kubernetes.io/projected/1a6c450e-f2fc-44cd-9e77-caf329de70e7-kube-api-access-jrngq\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-kubernetes\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-tmp\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cnibin\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cnibin\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-k8s-cni-cncf-io\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsdl\" (UniqueName: \"kubernetes.io/projected/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-kube-api-access-cnsdl\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:28.512608 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmcgf\" (UniqueName: \"kubernetes.io/projected/e72130d2-cf4b-4cb7-9912-a7fe11007acd-kube-api-access-vmcgf\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-k8s-cni-cncf-io\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-binary-copy\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-system-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-etc-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-tuned\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-system-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-device-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-env-overrides\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-dbus\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-device-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-systemd\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-sys\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.513411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-sys-fs\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-agent-certs\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-bin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-etc-kubernetes\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-ovn\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.512995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513003 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-bin\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-run\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-multus-certs\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-log-socket\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkxp\" (UniqueName: \"kubernetes.io/projected/6da7c700-b7cc-4c87-9400-ef17a3557825-kube-api-access-pkkxp\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-multus\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-sys-fs\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-etc-kubernetes\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-kubelet\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-daemon-config\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-slash\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-netns\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513251 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-cni-multus\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-kubelet-config\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-multus-certs\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-var-lib-kubelet\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-konnectivity-ca\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkp5c\" (UniqueName: \"kubernetes.io/projected/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-kube-api-access-fkp5c\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-socket-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ctn\" (UniqueName: \"kubernetes.io/projected/0dd72a65-aeeb-4cef-844d-0277bfde3433-kube-api-access-89ctn\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e72130d2-cf4b-4cb7-9912-a7fe11007acd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-netns\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-socket-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-hostroot\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-host-run-netns\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.514555 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-systemd-units\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-hostroot\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-var-lib-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-var-lib-kubelet\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5c4960-f167-4d58-999e-c5945302ede8-iptables-alerter-script\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-netd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5c4960-f167-4d58-999e-c5945302ede8-host-slash\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-daemon-config\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-system-cni-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-conf-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-config\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513818 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e72130d2-cf4b-4cb7-9912-a7fe11007acd-system-cni-dir\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovn-node-metrics-cert\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-conf-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw7z\" (UniqueName: \"kubernetes.io/projected/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-kube-api-access-fbw7z\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.515083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513902 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dd72a65-aeeb-4cef-844d-0277bfde3433-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.515563 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.513920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-multus-cni-dir\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.519293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.519270 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 20 01:52:28.523002 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.522980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw7z\" (UniqueName: \"kubernetes.io/projected/ee537002-c5d1-4dcf-99ae-13e1b0fe1607-kube-api-access-fbw7z\") pod \"node-ca-dczpm\" (UID: \"ee537002-c5d1-4dcf-99ae-13e1b0fe1607\") " pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.523108 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.522982 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmcgf\" (UniqueName: \"kubernetes.io/projected/e72130d2-cf4b-4cb7-9912-a7fe11007acd-kube-api-access-vmcgf\") pod \"multus-additional-cni-plugins-89dck\" (UID: \"e72130d2-cf4b-4cb7-9912-a7fe11007acd\") " pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.523108 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.523097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsdl\" (UniqueName: \"kubernetes.io/projected/7c37fd70-4ebb-47ba-ba4e-6a39e150e037-kube-api-access-cnsdl\") pod \"multus-tz2wh\" (UID: \"7c37fd70-4ebb-47ba-ba4e-6a39e150e037\") " pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.523902 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.523882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ctn\" (UniqueName: \"kubernetes.io/projected/0dd72a65-aeeb-4cef-844d-0277bfde3433-kube-api-access-89ctn\") pod \"aws-ebs-csi-driver-node-kvgtf\" (UID: \"0dd72a65-aeeb-4cef-844d-0277bfde3433\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.525046 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.525004 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:28.534112 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.534090 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1391da81b03a052e00ea4b68a886b3f.slice/crio-b23bb72dfa0b6fd474dab0d0eb4a265699a2dbe808d12e63327b93efa33fbf90 WatchSource:0}: Error finding container b23bb72dfa0b6fd474dab0d0eb4a265699a2dbe808d12e63327b93efa33fbf90: Status 404 returned error can't find the container with id b23bb72dfa0b6fd474dab0d0eb4a265699a2dbe808d12e63327b93efa33fbf90 Sep 20 01:52:28.534281 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.534269 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc472519add22672739b1e4cfa2793bb7.slice/crio-41ee819b9603bcf53f443691386c411792ce00a9cf2d2e60e0ef569cbebd2565 WatchSource:0}: Error finding container 41ee819b9603bcf53f443691386c411792ce00a9cf2d2e60e0ef569cbebd2565: Status 404 returned error can't find the container with id 41ee819b9603bcf53f443691386c411792ce00a9cf2d2e60e0ef569cbebd2565 Sep 20 01:52:28.542883 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.542863 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 01:52:28.614425 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.614425 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-modprobe-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-conf\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-node-log\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-script-lib\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysconfig\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8q2f\" (UniqueName: \"kubernetes.io/projected/2c5c4960-f167-4d58-999e-c5945302ede8-kube-api-access-b8q2f\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614539 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-node-log\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614591 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-modprobe-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614600 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-conf\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614600 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysconfig\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-host\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-host\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.614661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-lib-modules\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da7c700-b7cc-4c87-9400-ef17a3557825-hosts-file\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-kubelet\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614722 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6da7c700-b7cc-4c87-9400-ef17a3557825-hosts-file\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-systemd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614756 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-kubelet\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da7c700-b7cc-4c87-9400-ef17a3557825-tmp-dir\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-systemd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-lib-modules\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mk74\" (UniqueName: \"kubernetes.io/projected/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-kube-api-access-7mk74\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-bin\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrngq\" (UniqueName: \"kubernetes.io/projected/1a6c450e-f2fc-44cd-9e77-caf329de70e7-kube-api-access-jrngq\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-kubernetes\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-tmp\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615178 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-bin\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-etc-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-script-lib\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da7c700-b7cc-4c87-9400-ef17a3557825-tmp-dir\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-kubernetes\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-etc-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.615368 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-tuned\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.614980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.615444 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:29.115403442 +0000 UTC m=+2.128600965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-sysctl-d\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-env-overrides\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.615837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-dbus\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-systemd\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-sys\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.615551 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-agent-certs\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-ovn\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.615600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:29.115584331 +0000 UTC m=+2.128781859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-systemd\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-run\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-dbus\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-sys\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-log-socket\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-run-ovn\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkxp\" (UniqueName: \"kubernetes.io/projected/6da7c700-b7cc-4c87-9400-ef17a3557825-kube-api-access-pkkxp\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-log-socket\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615778 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-run\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615826 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-slash\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-env-overrides\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.616632 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-slash\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.615996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-netns\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-kubelet-config\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-konnectivity-ca\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-run-netns\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe772d7-ed21-4135-ac2d-a2922c120101-kubelet-config\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp5c\" (UniqueName: \"kubernetes.io/projected/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-kube-api-access-fkp5c\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-systemd-units\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-systemd-units\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-var-lib-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-var-lib-kubelet\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5c4960-f167-4d58-999e-c5945302ede8-iptables-alerter-script\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-netd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5c4960-f167-4d58-999e-c5945302ede8-host-slash\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-config\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovn-node-metrics-cert\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-konnectivity-ca\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.617225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5c4960-f167-4d58-999e-c5945302ede8-host-slash\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.617674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-host-cni-netd\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5c4960-f167-4d58-999e-c5945302ede8-iptables-alerter-script\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.617674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-var-lib-kubelet\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.617674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616942 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a6c450e-f2fc-44cd-9e77-caf329de70e7-var-lib-openvswitch\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.617674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.616914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovnkube-config\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.618000 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.617981 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-etc-tuned\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.618112 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.618004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-tmp\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.618365 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.618349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b4698d-5f2b-4a74-8a2f-fe9a30773c74-agent-certs\") pod \"konnectivity-agent-v24kh\" (UID: \"19b4698d-5f2b-4a74-8a2f-fe9a30773c74\") " pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.618672 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.618657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a6c450e-f2fc-44cd-9e77-caf329de70e7-ovn-node-metrics-cert\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.621917 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.621894 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:28.622053 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.621921 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:28.622053 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.621936 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:28.622053 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:28.622009 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:29.121990523 +0000 UTC m=+2.135188050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:28.623084 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.623062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mk74\" (UniqueName: \"kubernetes.io/projected/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-kube-api-access-7mk74\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:28.623370 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.623357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8q2f\" (UniqueName: \"kubernetes.io/projected/2c5c4960-f167-4d58-999e-c5945302ede8-kube-api-access-b8q2f\") pod \"iptables-alerter-pkjw4\" (UID: \"2c5c4960-f167-4d58-999e-c5945302ede8\") " pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:28.623410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.623387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrngq\" (UniqueName: \"kubernetes.io/projected/1a6c450e-f2fc-44cd-9e77-caf329de70e7-kube-api-access-jrngq\") pod \"ovnkube-node-7b7rr\" (UID: \"1a6c450e-f2fc-44cd-9e77-caf329de70e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.623972 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.623956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp5c\" (UniqueName: \"kubernetes.io/projected/b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c-kube-api-access-fkp5c\") pod \"tuned-dd47k\" (UID: \"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.624271 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.624252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkxp\" (UniqueName: \"kubernetes.io/projected/6da7c700-b7cc-4c87-9400-ef17a3557825-kube-api-access-pkkxp\") pod \"node-resolver-xpl5v\" (UID: \"6da7c700-b7cc-4c87-9400-ef17a3557825\") " pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:28.717941 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.717861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" Sep 20 01:52:28.728129 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.728104 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd72a65_aeeb_4cef_844d_0277bfde3433.slice/crio-6dbf9c46d5eb9c8f1526419919e3f273eb3d0347859b0b756a8b8070bf6ac224 WatchSource:0}: Error finding container 6dbf9c46d5eb9c8f1526419919e3f273eb3d0347859b0b756a8b8070bf6ac224: Status 404 returned error can't find the container with id 6dbf9c46d5eb9c8f1526419919e3f273eb3d0347859b0b756a8b8070bf6ac224 Sep 20 01:52:28.744972 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.744953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dczpm" Sep 20 01:52:28.750751 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.750732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89dck" Sep 20 01:52:28.752634 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.752610 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee537002_c5d1_4dcf_99ae_13e1b0fe1607.slice/crio-a24bbfa8277eda265858df9d0a94251079a7a9c292fc1f83e886eead1d48f61f WatchSource:0}: Error finding container a24bbfa8277eda265858df9d0a94251079a7a9c292fc1f83e886eead1d48f61f: Status 404 returned error can't find the container with id a24bbfa8277eda265858df9d0a94251079a7a9c292fc1f83e886eead1d48f61f Sep 20 01:52:28.759439 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.759417 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72130d2_cf4b_4cb7_9912_a7fe11007acd.slice/crio-f429f87a7e1958c9ef9bd786ede35f68a74365aa1601c97ab5d2f0fcd9e87942 WatchSource:0}: Error finding container f429f87a7e1958c9ef9bd786ede35f68a74365aa1601c97ab5d2f0fcd9e87942: Status 404 returned error can't find the container with id f429f87a7e1958c9ef9bd786ede35f68a74365aa1601c97ab5d2f0fcd9e87942 Sep 20 01:52:28.772242 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.772219 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tz2wh" Sep 20 01:52:28.777222 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.777205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:28.780604 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.780575 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c37fd70_4ebb_47ba_ba4e_6a39e150e037.slice/crio-8f5bf38592689df5817cd9232ba8f4dff167ee19fa195e3c1983d6c26aa76dba WatchSource:0}: Error finding container 8f5bf38592689df5817cd9232ba8f4dff167ee19fa195e3c1983d6c26aa76dba: Status 404 returned error can't find the container with id 8f5bf38592689df5817cd9232ba8f4dff167ee19fa195e3c1983d6c26aa76dba Sep 20 01:52:28.786543 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.786526 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6c450e_f2fc_44cd_9e77_caf329de70e7.slice/crio-9de1ce3088e5d843a87d6bacb42f11e8554099795cc56110f8860f2a018650a1 WatchSource:0}: Error finding container 9de1ce3088e5d843a87d6bacb42f11e8554099795cc56110f8860f2a018650a1: Status 404 returned error can't find the container with id 9de1ce3088e5d843a87d6bacb42f11e8554099795cc56110f8860f2a018650a1 Sep 20 01:52:28.792999 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.792980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:28.802095 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.802064 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b4698d_5f2b_4a74_8a2f_fe9a30773c74.slice/crio-ea128346e3ee183532fe8486359d9aafffbb1d4672920379c968dca85ea3ac03 WatchSource:0}: Error finding container ea128346e3ee183532fe8486359d9aafffbb1d4672920379c968dca85ea3ac03: Status 404 returned error can't find the container with id ea128346e3ee183532fe8486359d9aafffbb1d4672920379c968dca85ea3ac03 Sep 20 01:52:28.818901 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.818877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dd47k" Sep 20 01:52:28.826744 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:28.826721 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0fe5d_78a6_4176_8a06_3f5e3abbbe4c.slice/crio-4a494c05ecc2096b36cdd435d032f55a6e99ff3870f0de49219a8afc45fec251 WatchSource:0}: Error finding container 4a494c05ecc2096b36cdd435d032f55a6e99ff3870f0de49219a8afc45fec251: Status 404 returned error can't find the container with id 4a494c05ecc2096b36cdd435d032f55a6e99ff3870f0de49219a8afc45fec251 Sep 20 01:52:28.845831 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:28.845809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpl5v" Sep 20 01:52:30.347521 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.347504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkjw4" Sep 20 01:52:30.349856 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:30.349835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da7c700_b7cc_4c87_9400_ef17a3557825.slice/crio-2a382a188921e8be17f5e8a4044ccdc93f25ccbb76dada8f7d5bdc6c2e707e08 WatchSource:0}: Error finding container 2a382a188921e8be17f5e8a4044ccdc93f25ccbb76dada8f7d5bdc6c2e707e08: Status 404 returned error can't find the container with id 2a382a188921e8be17f5e8a4044ccdc93f25ccbb76dada8f7d5bdc6c2e707e08 Sep 20 01:52:30.356291 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:52:30.356271 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5c4960_f167_4d58_999e_c5945302ede8.slice/crio-d07ffa5b588dab74deff98d7d5b96c51a200049c8929e7052f5b7b432f454c00 WatchSource:0}: Error finding container d07ffa5b588dab74deff98d7d5b96c51a200049c8929e7052f5b7b432f454c00: Status 404 returned error can't find the container with id d07ffa5b588dab74deff98d7d5b96c51a200049c8929e7052f5b7b432f454c00 Sep 20 01:52:30.406450 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.406419 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.615125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.615171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.615290 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.615291 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.615347 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:31.615329598 +0000 UTC m=+3.133645346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:30.615509 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.615364 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:31.615354654 +0000 UTC m=+3.133670395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:30.715576 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.715546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:30.715827 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.715795 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:30.715981 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.715831 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:30.715981 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.715845 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:30.715981 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:30.715918 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:31.715885704 +0000 UTC m=+3.234201452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:30.943303 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.943212 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-19 01:47:29 +0000 UTC" deadline="2028-05-02 18:48:44.465629405 +0000 UTC" Sep 20 01:52:30.943303 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:30.943254 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14176h56m13.522380047s" Sep 20 01:52:31.011500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.011465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:31.011685 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.011608 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:31.030103 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.030051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tz2wh" event={"ID":"7c37fd70-4ebb-47ba-ba4e-6a39e150e037","Type":"ContainerStarted","Data":"8f5bf38592689df5817cd9232ba8f4dff167ee19fa195e3c1983d6c26aa76dba"} Sep 20 01:52:31.040317 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.040283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerStarted","Data":"f429f87a7e1958c9ef9bd786ede35f68a74365aa1601c97ab5d2f0fcd9e87942"} Sep 20 01:52:31.051891 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.051859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dczpm" event={"ID":"ee537002-c5d1-4dcf-99ae-13e1b0fe1607","Type":"ContainerStarted","Data":"a24bbfa8277eda265858df9d0a94251079a7a9c292fc1f83e886eead1d48f61f"} Sep 20 01:52:31.070341 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.070259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" event={"ID":"0dd72a65-aeeb-4cef-844d-0277bfde3433","Type":"ContainerStarted","Data":"6dbf9c46d5eb9c8f1526419919e3f273eb3d0347859b0b756a8b8070bf6ac224"} Sep 20 01:52:31.111801 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.111691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v24kh" event={"ID":"19b4698d-5f2b-4a74-8a2f-fe9a30773c74","Type":"ContainerStarted","Data":"ea128346e3ee183532fe8486359d9aafffbb1d4672920379c968dca85ea3ac03"} Sep 20 01:52:31.117157 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.116982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" event={"ID":"e1391da81b03a052e00ea4b68a886b3f","Type":"ContainerStarted","Data":"b23bb72dfa0b6fd474dab0d0eb4a265699a2dbe808d12e63327b93efa33fbf90"} Sep 20 01:52:31.124937 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.124896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" event={"ID":"c472519add22672739b1e4cfa2793bb7","Type":"ContainerStarted","Data":"41ee819b9603bcf53f443691386c411792ce00a9cf2d2e60e0ef569cbebd2565"} Sep 20 01:52:31.143297 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.143266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkjw4" event={"ID":"2c5c4960-f167-4d58-999e-c5945302ede8","Type":"ContainerStarted","Data":"d07ffa5b588dab74deff98d7d5b96c51a200049c8929e7052f5b7b432f454c00"} Sep 20 01:52:31.148181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.148151 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpl5v" event={"ID":"6da7c700-b7cc-4c87-9400-ef17a3557825","Type":"ContainerStarted","Data":"2a382a188921e8be17f5e8a4044ccdc93f25ccbb76dada8f7d5bdc6c2e707e08"} Sep 20 01:52:31.171563 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.171408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dd47k" event={"ID":"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c","Type":"ContainerStarted","Data":"4a494c05ecc2096b36cdd435d032f55a6e99ff3870f0de49219a8afc45fec251"} Sep 20 01:52:31.181965 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.181927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"9de1ce3088e5d843a87d6bacb42f11e8554099795cc56110f8860f2a018650a1"} Sep 20 01:52:31.326746 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.326677 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:31.623016 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.622928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:31.623274 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.623041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:31.623274 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.623170 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:31.623274 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.623233 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:33.623216066 +0000 UTC m=+5.141531815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:31.623687 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.623663 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:31.623811 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.623719 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:33.623702272 +0000 UTC m=+5.142018053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:31.723589 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.723552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:31.723791 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.723773 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:31.723854 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.723798 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:31.723854 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.723810 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:31.723979 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:31.723871 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:33.723852204 +0000 UTC m=+5.242167983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:31.943629 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.943520 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-19 01:47:29 +0000 UTC" deadline="2028-05-03 03:04:56.880901702 +0000 UTC" Sep 20 01:52:31.943629 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:31.943582 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14185h12m24.937324102s" Sep 20 01:52:32.010034 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:32.009833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:32.010034 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:32.009965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:32.010377 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:32.010333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:32.010437 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:32.010420 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:32.420607 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:32.420526 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 20 01:52:33.012845 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:33.011985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:33.012845 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.012413 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:33.639630 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:33.639593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:33.639630 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:33.639648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:33.639880 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.639778 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:33.639880 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.639842 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:37.639823454 +0000 UTC m=+9.158139192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:33.640261 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.640242 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:33.640319 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.640298 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:37.64028336 +0000 UTC m=+9.158599100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:33.740879 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:33.740844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:33.741090 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.741069 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:33.741169 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.741098 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:33.741169 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.741113 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:33.741270 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:33.741175 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:37.741156612 +0000 UTC m=+9.259472351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:34.009195 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:34.009118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:34.009358 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:34.009248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:34.009623 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:34.009605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:34.009718 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:34.009700 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:35.011354 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:35.011324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:35.011791 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:35.011459 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:36.009937 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:36.009735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:36.009937 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:36.009878 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:36.009937 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:36.009936 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:36.010196 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:36.010009 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:37.010687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:37.010189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:37.010687 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.010327 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:37.673783 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:37.673737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:37.673975 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:37.673795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:37.673975 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.673965 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:37.674098 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.674032 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:45.674013203 +0000 UTC m=+17.192328957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:37.674441 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.674421 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:37.674570 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.674478 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:45.674463197 +0000 UTC m=+17.192778945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:37.775111 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:37.774399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:37.775111 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.774615 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:37.775111 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.774635 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:37.775111 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.774649 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:37.775111 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:37.774709 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:52:45.774690897 +0000 UTC m=+17.293006645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:38.009336 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:38.009120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:38.009336 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:38.009241 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:38.009662 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:38.009608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:38.009745 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:38.009689 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:39.010013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:39.009983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:39.010440 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:39.010076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:40.009436 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:40.009394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:40.009634 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:40.009415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:40.009634 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:40.009528 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:40.009634 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:40.009592 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:41.009828 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:41.009790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:41.010377 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:41.009955 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:42.009301 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:42.009060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:42.009301 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:42.009170 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:42.009301 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:42.009213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:42.009301 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:42.009272 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:43.009278 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:43.009240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:43.009740 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:43.009400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:44.009070 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:44.009040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:44.009240 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:44.009081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:44.009240 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:44.009180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:44.009587 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:44.009324 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:45.009775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:45.009745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:45.010164 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.009882 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:45.734048 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:45.734007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:45.734048 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:45.734052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:45.734329 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.734169 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:45.734329 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.734181 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:45.734329 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.734240 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:01.73422069 +0000 UTC m=+33.252536432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:52:45.734329 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.734255 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:01.734248362 +0000 UTC m=+33.252564098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:52:45.835050 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:45.835023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:45.835263 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.835162 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:52:45.835263 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.835176 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:52:45.835263 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.835187 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:45.835263 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:45.835243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:01.835230138 +0000 UTC m=+33.353545896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:52:46.009659 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:46.009579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:46.009799 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:46.009590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:46.009799 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:46.009714 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:46.009799 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:46.009787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:47.009605 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:47.009569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:47.009788 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:47.009687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:48.009560 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:48.009523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:48.010040 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:48.009529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:48.010040 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:48.009660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:48.010040 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:48.009775 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:49.009893 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.009726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:49.010514 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:49.009983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:49.251575 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.251159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dd47k" event={"ID":"b2f0fe5d-78a6-4176-8a06-3f5e3abbbe4c","Type":"ContainerStarted","Data":"c489ae0318ac6743da776a8edc135a0143c8c0921e7d6ae74ec3902d1187a98c"} Sep 20 01:52:49.257617 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257365 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:52:49.257743 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257701 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a6c450e-f2fc-44cd-9e77-caf329de70e7" containerID="c603c8f16a6c1dd85eaf20738f37b962846598f56058a2a44f8c3977c6bd6e21" exitCode=1 Sep 20 01:52:49.257807 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"c217487d3f94f7fd26da3b2fa1ecb156d9668481ece56c5d8d1e084add19b85d"} Sep 20 01:52:49.257861 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"542c08f444adc2c2735214a07467faceaf106462e87550d1c9fe55dbc41908b7"} Sep 20 01:52:49.257938 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerDied","Data":"c603c8f16a6c1dd85eaf20738f37b962846598f56058a2a44f8c3977c6bd6e21"} Sep 20 01:52:49.258000 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.257961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"cb8b24569d2ac65f79eb0effb787c3da04265a0f97664cd2fc8eff7683a0cd14"} Sep 20 01:52:49.259936 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.259634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tz2wh" event={"ID":"7c37fd70-4ebb-47ba-ba4e-6a39e150e037","Type":"ContainerStarted","Data":"f31c9564724068487a81303dc0b25a5850948d26fa9251213041eaabdbafb4bf"} Sep 20 01:52:49.265103 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.265050 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" event={"ID":"c472519add22672739b1e4cfa2793bb7","Type":"ContainerStarted","Data":"5b93cd4d1a30aa2cbbddef1cbcb716e841acd269d273ad0ac6a9500c69fefe97"} Sep 20 01:52:49.269118 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.269064 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dd47k" podStartSLOduration=1.947372132 podStartE2EDuration="20.269051212s" totalImagesPullingTime="18.32167908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:49.268809077 +0000 UTC m=+20.787124835" watchObservedRunningTime="2026-09-20 01:52:49.269051212 +0000 UTC m=+20.787366965" Sep 20 01:52:49.298037 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:49.297984 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-50.ec2.internal" podStartSLOduration=20.2979627 podStartE2EDuration="20.2979627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:49.281833061 +0000 UTC m=+20.800148820" watchObservedRunningTime="2026-09-20 01:52:49.2979627 +0000 UTC m=+20.816278457" Sep 20 01:52:50.009547 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.009284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:50.009706 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.009284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:50.009706 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:50.009647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:50.009706 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:50.009647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:50.269818 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.269705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkjw4" event={"ID":"2c5c4960-f167-4d58-999e-c5945302ede8","Type":"ContainerStarted","Data":"1c2218e7a2152bb9bab761adc35f6a7d77f4a19f3d7895a457f29e7d365f7af9"} Sep 20 01:52:50.271420 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.271389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpl5v" event={"ID":"6da7c700-b7cc-4c87-9400-ef17a3557825","Type":"ContainerStarted","Data":"44d6a100de6ec3b70b35304965545de5c60682341b3728d2747df6832c96690b"} Sep 20 01:52:50.274797 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.274780 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:52:50.275220 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.275201 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"69efb99fb0b34f3bac566b290de4149fc5a97f0f11eaab5237086ab7c88b59bc"} Sep 20 01:52:50.275334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.275273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"6eedd2ed62859d013e4b424ff44401c7d801e55c2611bd53c1acf82f31e20d44"} Sep 20 01:52:50.277049 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.277028 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="62954868132d122aad2d65e1e021012281c9d7592054b994c3159c89fa70bf70" exitCode=0 Sep 20 01:52:50.277163 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.277097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"62954868132d122aad2d65e1e021012281c9d7592054b994c3159c89fa70bf70"} Sep 20 01:52:50.278951 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.278927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dczpm" event={"ID":"ee537002-c5d1-4dcf-99ae-13e1b0fe1607","Type":"ContainerStarted","Data":"834d26797cd4e607d048366ef584c905ec9f7f4c622d1a83e584930c3780500a"} Sep 20 01:52:50.280557 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.280507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" event={"ID":"0dd72a65-aeeb-4cef-844d-0277bfde3433","Type":"ContainerStarted","Data":"8f93403de45b9ad0e0570f5c0d0e0421797b58b91b9d7b9b0810dcbb4a915e3f"} Sep 20 01:52:50.282193 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.282174 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v24kh" event={"ID":"19b4698d-5f2b-4a74-8a2f-fe9a30773c74","Type":"ContainerStarted","Data":"be8827c7b55200b9c55e0a65ba618f0bd7c9d96456d044a070b693dcc7c24b24"} Sep 20 01:52:50.284131 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.284105 2578 generic.go:356] "Generic (PLEG): container finished" podID="e1391da81b03a052e00ea4b68a886b3f" containerID="f3c40147329e382681766ad57eb0c244ecb92a4328285109e0ffc9dcd59a5d01" exitCode=0 Sep 20 01:52:50.284225 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.284206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" event={"ID":"e1391da81b03a052e00ea4b68a886b3f","Type":"ContainerDied","Data":"f3c40147329e382681766ad57eb0c244ecb92a4328285109e0ffc9dcd59a5d01"} Sep 20 01:52:50.284875 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.284833 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tz2wh" podStartSLOduration=2.4522815319999998 podStartE2EDuration="21.284817953s" totalImagesPullingTime="18.832536421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:49.298706877 +0000 UTC m=+20.817022636" watchObservedRunningTime="2026-09-20 01:52:50.284817953 +0000 UTC m=+21.803133709" Sep 20 01:52:50.285416 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.285380 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkjw4" podStartSLOduration=3.018292849 podStartE2EDuration="21.285369859s" totalImagesPullingTime="18.26707701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:50.28473338 +0000 UTC m=+21.803049139" watchObservedRunningTime="2026-09-20 01:52:50.285369859 +0000 UTC m=+21.803685615" Sep 20 01:52:50.298455 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.298410 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xpl5v" podStartSLOduration=3.025220049 podStartE2EDuration="21.298395446s" totalImagesPullingTime="18.273175397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:50.297861 +0000 UTC m=+21.816176755" watchObservedRunningTime="2026-09-20 01:52:50.298395446 +0000 UTC m=+21.816711203" Sep 20 01:52:50.323691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.323640 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dczpm" podStartSLOduration=2.948087806 podStartE2EDuration="21.323625131s" totalImagesPullingTime="18.375537325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:50.323098536 +0000 UTC m=+21.841414303" watchObservedRunningTime="2026-09-20 01:52:50.323625131 +0000 UTC m=+21.841940888" Sep 20 01:52:50.336494 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.336451 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v24kh" podStartSLOduration=2.989091139 podStartE2EDuration="21.336433484s" totalImagesPullingTime="18.347342345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:50.336392968 +0000 UTC m=+21.854708724" watchObservedRunningTime="2026-09-20 01:52:50.336433484 +0000 UTC m=+21.854749240" Sep 20 01:52:50.716432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.716401 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 20 01:52:50.976479 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.976365 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-20T01:52:50.716425355Z","UUID":"88d80ec0-59c2-4b4c-92dd-3d745c16c7b8","Handler":null,"Name":"","Endpoint":""} Sep 20 01:52:50.977954 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.977930 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 20 01:52:50.978075 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:50.977967 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 20 01:52:51.009603 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:51.009565 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:51.009757 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:51.009694 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:51.289054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:51.288935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" event={"ID":"e1391da81b03a052e00ea4b68a886b3f","Type":"ContainerStarted","Data":"e5f91df2927580710cae887c4a65d0c7835c7b0098a64cf4bf0000ee8df29916"} Sep 20 01:52:51.291062 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:51.291002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" event={"ID":"0dd72a65-aeeb-4cef-844d-0277bfde3433","Type":"ContainerStarted","Data":"2ad2b9df2a84b463231679b3441d359bdcc4d7b6d27740fde66b5a89497bfc85"} Sep 20 01:52:51.304064 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:51.304008 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-50.ec2.internal" podStartSLOduration=22.303989113 podStartE2EDuration="22.303989113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:51.30372446 +0000 UTC m=+22.822040217" watchObservedRunningTime="2026-09-20 01:52:51.303989113 +0000 UTC m=+22.822304871" Sep 20 01:52:52.010031 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.009830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:52.010255 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.009830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:52.010255 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:52.010116 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:52.010255 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:52.010179 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:52.296055 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.295955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" event={"ID":"0dd72a65-aeeb-4cef-844d-0277bfde3433","Type":"ContainerStarted","Data":"c76b320deeb6d5edf65deb2642a5467ca1f12ef058bb3e88882d5a189221cfa8"} Sep 20 01:52:52.300129 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.300103 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:52:52.300532 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.300460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"ba41b1bece8b5755a7dbbb8043466f405aef350e1a31010372a58429d21cbdf7"} Sep 20 01:52:52.312569 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:52.312519 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvgtf" podStartSLOduration=2.586953933 podStartE2EDuration="23.312499862s" totalImagesPullingTime="20.725545929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:52.311695785 +0000 UTC m=+23.830011541" watchObservedRunningTime="2026-09-20 01:52:52.312499862 +0000 UTC m=+23.830815621" Sep 20 01:52:53.010061 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:53.010024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:53.010282 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:53.010159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:54.010012 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:54.009967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:54.010012 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:54.010002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:54.010503 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:54.010093 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:54.010503 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:54.010225 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:54.118669 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:54.118629 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:54.119371 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:54.119350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:55.009516 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.009339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:55.009700 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:55.009614 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:55.309749 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.309677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:52:55.310230 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.310039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"e06918eab35cfb177df6600a432252318689691b3f123189adeda61ed6d942dd"} Sep 20 01:52:55.310429 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.310316 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:55.310429 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.310386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:55.310599 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.310499 2578 scope.go:117] "RemoveContainer" containerID="c603c8f16a6c1dd85eaf20738f37b962846598f56058a2a44f8c3977c6bd6e21" Sep 20 01:52:55.311896 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.311873 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="c9e1ab7cb11577fdc9bca61bbe31e86cab80d0115c1160d864e460bc1d050f1e" exitCode=0 Sep 20 01:52:55.311993 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.311936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"c9e1ab7cb11577fdc9bca61bbe31e86cab80d0115c1160d864e460bc1d050f1e"} Sep 20 01:52:55.328690 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:55.328673 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:56.009863 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.009825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:56.010045 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.009835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:56.010045 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:56.009990 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:56.010170 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:56.010126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:56.254108 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.254072 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp4rc"] Sep 20 01:52:56.257096 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.257070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5j9ht"] Sep 20 01:52:56.257676 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.257654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwfcm"] Sep 20 01:52:56.257932 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.257776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:56.258094 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:56.257897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:56.317567 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.317499 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:52:56.318314 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.317923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" event={"ID":"1a6c450e-f2fc-44cd-9e77-caf329de70e7","Type":"ContainerStarted","Data":"35437178a9c27058f5fb7d66cf6ea16e5453eb70badcc39e90d939a31c4f2d63"} Sep 20 01:52:56.318314 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.318086 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:56.320169 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.320139 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="b03c9a8774b248844eb185d3022bf492861a4a859b0aea74b69d8fbb17ede75b" exitCode=0 Sep 20 01:52:56.320293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.320188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:56.320293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.320208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"b03c9a8774b248844eb185d3022bf492861a4a859b0aea74b69d8fbb17ede75b"} Sep 20 01:52:56.320411 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:56.320293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:56.320411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.320348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:56.320510 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:56.320428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:56.334477 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.334459 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:52:56.344107 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:56.344069 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" podStartSLOduration=8.967301122 podStartE2EDuration="27.344058182s" totalImagesPullingTime="18.37675706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:52:56.343165073 +0000 UTC m=+27.861480829" watchObservedRunningTime="2026-09-20 01:52:56.344058182 +0000 UTC m=+27.862373933" Sep 20 01:52:57.324239 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:57.324140 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="fdb8cc27c9f04c83acef03627f1330ad210958c8866d3dcb1668d8d5c388f40d" exitCode=0 Sep 20 01:52:57.324239 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:57.324229 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"fdb8cc27c9f04c83acef03627f1330ad210958c8866d3dcb1668d8d5c388f40d"} Sep 20 01:52:58.009654 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.009334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:52:58.009803 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.009371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:52:58.009803 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:58.009683 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:52:58.009803 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.009402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:52:58.009803 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:58.009763 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:52:58.009999 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:52:58.009846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:52:58.019796 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.019771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:52:58.019948 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.019925 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 20 01:52:58.021499 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:52:58.020553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v24kh" Sep 20 01:53:00.009769 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:00.009736 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:00.010219 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:00.009856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp4rc" podUID="5ddc7b08-635c-421f-856b-93f530c903b9" Sep 20 01:53:00.010219 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:00.009932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:00.010219 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:00.010046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5j9ht" podUID="cfe772d7-ed21-4135-ac2d-a2922c120101" Sep 20 01:53:00.010330 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:00.010299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:00.010437 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:00.010416 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwfcm" podUID="70e1da5e-94ec-46be-85e8-dc0ced0ff4a4" Sep 20 01:53:01.748982 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.748955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:01.749327 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.748989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:01.749327 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.749091 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:53:01.749327 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.749095 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 20 01:53:01.749327 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.749143 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs podName:70e1da5e-94ec-46be-85e8-dc0ced0ff4a4 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:33.749129144 +0000 UTC m=+65.267444880 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs") pod "network-metrics-daemon-xwfcm" (UID: "70e1da5e-94ec-46be-85e8-dc0ced0ff4a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 20 01:53:01.749327 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.749155 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret podName:cfe772d7-ed21-4135-ac2d-a2922c120101 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:33.749149325 +0000 UTC m=+65.267465061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret") pod "global-pull-secret-syncer-5j9ht" (UID: "cfe772d7-ed21-4135-ac2d-a2922c120101") : object "kube-system"/"original-pull-secret" not registered Sep 20 01:53:01.798001 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.797972 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-50.ec2.internal" event="NodeReady" Sep 20 01:53:01.798176 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.798083 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 20 01:53:01.849543 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.849500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:01.850393 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.849700 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 20 01:53:01.850393 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.849718 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 20 01:53:01.850393 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.849730 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wlgc8 for pod openshift-network-diagnostics/network-check-target-pp4rc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:53:01.850393 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:01.849781 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8 podName:5ddc7b08-635c-421f-856b-93f530c903b9 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:33.849765703 +0000 UTC m=+65.368081458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlgc8" (UniqueName: "kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8") pod "network-check-target-pp4rc" (UID: "5ddc7b08-635c-421f-856b-93f530c903b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 20 01:53:01.850393 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.850132 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wdsjv"] Sep 20 01:53:01.868413 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.868391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:01.868993 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.868283 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:53:01.871410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.871133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 20 01:53:01.871410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.871135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 20 01:53:01.871410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.871255 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp8gx\"" Sep 20 01:53:01.882342 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.882320 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xvbhm"] Sep 20 01:53:01.882480 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.882456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.884753 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.884736 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9d2x\"" Sep 20 01:53:01.884922 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.884756 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 20 01:53:01.884922 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.884807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 20 01:53:01.885107 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.885092 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 20 01:53:01.889354 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.889338 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 20 01:53:01.904448 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.904423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvbhm"] Sep 20 01:53:01.904448 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.904451 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wdsjv"] Sep 20 01:53:01.904596 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.904467 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:53:01.904596 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.904545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:01.906800 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.906768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w9zwc\"" Sep 20 01:53:01.906895 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.906797 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 20 01:53:01.906962 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.906893 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 20 01:53:01.906962 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.906942 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 20 01:53:01.950646 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9d6\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.950755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420afa4e-3ba4-442c-8548-70cecec2bc11-tmp-dir\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:01.950755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:01.950755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hk4\" (UniqueName: \"kubernetes.io/projected/420afa4e-3ba4-442c-8548-70cecec2bc11-kube-api-access-82hk4\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:01.950755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.950971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.950971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420afa4e-3ba4-442c-8548-70cecec2bc11-config-volume\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:01.950971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.950971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.950971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.951188 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:01.951188 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:01.950996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.009105 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.009079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:02.009193 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.009178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:02.009246 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.009195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:02.011350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011326 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 20 01:53:02.011350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011339 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 20 01:53:02.011467 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011420 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 20 01:53:02.011614 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011601 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjlfs\"" Sep 20 01:53:02.011654 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011607 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvrtw\"" Sep 20 01:53:02.011828 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.011816 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 20 01:53:02.052195 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkn57\" (UniqueName: \"kubernetes.io/projected/2313a01c-8a54-4b2e-b113-6100216ce6b5-kube-api-access-hkn57\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.052317 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420afa4e-3ba4-442c-8548-70cecec2bc11-config-volume\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.052317 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052317 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052317 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.052295 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:02.052317 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.052304 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.052352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:02.552331876 +0000 UTC m=+34.070647615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9d6\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2313a01c-8a54-4b2e-b113-6100216ce6b5-cert\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.052796 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420afa4e-3ba4-442c-8548-70cecec2bc11-tmp-dir\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.052796 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.052796 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.052779 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:02.052957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/420afa4e-3ba4-442c-8548-70cecec2bc11-config-volume\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.052957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hk4\" (UniqueName: \"kubernetes.io/projected/420afa4e-3ba4-442c-8548-70cecec2bc11-kube-api-access-82hk4\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.052957 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.052834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:02.552817421 +0000 UTC m=+34.071133178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:02.052957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.052957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.053208 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.052971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.053294 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.053277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.053824 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.053803 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.056214 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.056192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.056292 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.056198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.060561 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.060539 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.060758 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.060743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9d6\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.070251 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.070229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420afa4e-3ba4-442c-8548-70cecec2bc11-tmp-dir\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.072060 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.072045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hk4\" (UniqueName: \"kubernetes.io/projected/420afa4e-3ba4-442c-8548-70cecec2bc11-kube-api-access-82hk4\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.153280 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.153246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkn57\" (UniqueName: \"kubernetes.io/projected/2313a01c-8a54-4b2e-b113-6100216ce6b5-kube-api-access-hkn57\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.153419 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.153340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2313a01c-8a54-4b2e-b113-6100216ce6b5-cert\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.155926 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.155893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2313a01c-8a54-4b2e-b113-6100216ce6b5-cert\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.161267 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.161245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkn57\" (UniqueName: \"kubernetes.io/projected/2313a01c-8a54-4b2e-b113-6100216ce6b5-kube-api-access-hkn57\") pod \"ingress-canary-xvbhm\" (UID: \"2313a01c-8a54-4b2e-b113-6100216ce6b5\") " pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.216661 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.216639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvbhm" Sep 20 01:53:02.339307 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.339125 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="c3abf87b1bc1cadd99aa1e8ce64340669db55f0fe45d92842a694f01f60dc578" exitCode=0 Sep 20 01:53:02.339307 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.339206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"c3abf87b1bc1cadd99aa1e8ce64340669db55f0fe45d92842a694f01f60dc578"} Sep 20 01:53:02.514549 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.514518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvbhm"] Sep 20 01:53:02.547797 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:02.547766 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2313a01c_8a54_4b2e_b113_6100216ce6b5.slice/crio-4d8e3f5247e903618674000cbcf956035df4a85070fcbf3d412ce1643469e5f9 WatchSource:0}: Error finding container 4d8e3f5247e903618674000cbcf956035df4a85070fcbf3d412ce1643469e5f9: Status 404 returned error can't find the container with id 4d8e3f5247e903618674000cbcf956035df4a85070fcbf3d412ce1643469e5f9 Sep 20 01:53:02.556491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.556471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:02.556583 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:02.556509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:02.556640 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.556617 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:02.556640 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.556635 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:02.556708 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.556647 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:02.556708 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.556687 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:03.556669322 +0000 UTC m=+35.074985058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:02.556708 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:02.556700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:03.556694061 +0000 UTC m=+35.075009796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:03.345938 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:03.345883 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72130d2-cf4b-4cb7-9912-a7fe11007acd" containerID="25454e3cbdbb2e366777eff27becd10d4f190ec61951ec1d610f2b44886a591e" exitCode=0 Sep 20 01:53:03.346489 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:03.345968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerDied","Data":"25454e3cbdbb2e366777eff27becd10d4f190ec61951ec1d610f2b44886a591e"} Sep 20 01:53:03.347438 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:03.347411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvbhm" event={"ID":"2313a01c-8a54-4b2e-b113-6100216ce6b5","Type":"ContainerStarted","Data":"4d8e3f5247e903618674000cbcf956035df4a85070fcbf3d412ce1643469e5f9"} Sep 20 01:53:03.564627 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:03.564589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:03.564765 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:03.564654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:03.564815 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:03.564800 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:03.564815 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:03.564812 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:03.564891 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:03.564865 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:05.564850773 +0000 UTC m=+37.083166522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:03.565304 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:03.565274 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:03.565421 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:03.565327 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:05.565311684 +0000 UTC m=+37.083627425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:04.354664 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:04.354630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89dck" event={"ID":"e72130d2-cf4b-4cb7-9912-a7fe11007acd","Type":"ContainerStarted","Data":"1c90e8a4220ab9ca28fe5e1015577a64affd0429ed363a55d55f07d042b172bf"} Sep 20 01:53:04.378570 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:04.378528 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-89dck" podStartSLOduration=7.364407926 podStartE2EDuration="35.378514255s" totalImagesPullingTime="28.014106329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:04.378235179 +0000 UTC m=+35.896550935" watchObservedRunningTime="2026-09-20 01:53:04.378514255 +0000 UTC m=+35.896830011" Sep 20 01:53:05.359011 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:05.358935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvbhm" event={"ID":"2313a01c-8a54-4b2e-b113-6100216ce6b5","Type":"ContainerStarted","Data":"1730471d7f2735555cf62971849818c8f0129108138645aead4fe4dcb6541383"} Sep 20 01:53:05.374272 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:05.374226 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xvbhm" podStartSLOduration=1.998998278 podStartE2EDuration="4.374211389s" totalImagesPullingTime="2.375213111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:05.373766214 +0000 UTC m=+36.892081970" watchObservedRunningTime="2026-09-20 01:53:05.374211389 +0000 UTC m=+36.892527145" Sep 20 01:53:05.582129 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:05.582090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:05.582285 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:05.582165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:05.582285 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:05.582239 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:05.582285 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:05.582256 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:05.582285 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:05.582265 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:05.582514 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:05.582314 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:09.582296235 +0000 UTC m=+41.100611973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:05.582514 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:05.582328 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:09.582321936 +0000 UTC m=+41.100637671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:09.612343 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:09.612159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:09.612361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:09.612314 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:09.612463 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:09.612467 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:17.612450549 +0000 UTC m=+49.130766284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:09.612473 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:09.612719 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:09.612509 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:17.612499392 +0000 UTC m=+49.130815127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:15.929996 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.929961 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-52f4v"] Sep 20 01:53:15.959485 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.959464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52f4v"] Sep 20 01:53:15.959615 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.959578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:15.961864 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.961839 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 20 01:53:15.961983 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.961848 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 20 01:53:15.962473 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.962456 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 20 01:53:15.962601 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.962482 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 20 01:53:15.962601 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:15.962516 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fcgnp\"" Sep 20 01:53:16.059275 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.059239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnct\" (UniqueName: \"kubernetes.io/projected/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-api-access-xhnct\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.059446 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.059285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-crio-socket\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.059446 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.059308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.059446 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.059391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-data-volume\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.059446 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.059439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.159949 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.159892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-data-volume\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.159979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.160016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnct\" (UniqueName: \"kubernetes.io/projected/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-api-access-xhnct\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.160037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-crio-socket\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.160051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160269 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.160126 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 20 01:53:16.160269 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.160143 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 20 01:53:16.160269 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.160201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:16.660187092 +0000 UTC m=+48.178502827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : secret "insights-runtime-extractor-tls" not found Sep 20 01:53:16.160269 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.160222 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:16.660207869 +0000 UTC m=+48.178523606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : configmap "kube-rbac-proxy" not found Sep 20 01:53:16.160269 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.160262 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-crio-socket\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.160421 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.160277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-data-volume\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.171917 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.171888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnct\" (UniqueName: \"kubernetes.io/projected/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-api-access-xhnct\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.664650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.664606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.664826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:16.664684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:16.664826 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.664746 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 20 01:53:16.664826 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.664783 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 20 01:53:16.664826 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.664826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:17.664803495 +0000 UTC m=+49.183119249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : configmap "kube-rbac-proxy" not found Sep 20 01:53:16.665090 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:16.664844 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:17.664835444 +0000 UTC m=+49.183151179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : secret "insights-runtime-extractor-tls" not found Sep 20 01:53:17.674372 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.674338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:17.674372 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.674375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.674404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.674440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674505 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674514 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674533 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674535 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674556 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbdb58748-hnvmj: secret "image-registry-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674580 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:19.674564332 +0000 UTC m=+51.192880067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : secret "insights-runtime-extractor-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls podName:1975626e-8e1c-4dbc-a3fb-a5eefc3805d2 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:33.674585907 +0000 UTC m=+65.192901664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls") pod "image-registry-5fbdb58748-hnvmj" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2") : secret "image-registry-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674612 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls podName:420afa4e-3ba4-442c-8548-70cecec2bc11 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:33.674606716 +0000 UTC m=+65.192922451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls") pod "dns-default-wdsjv" (UID: "420afa4e-3ba4-442c-8548-70cecec2bc11") : secret "dns-default-metrics-tls" not found Sep 20 01:53:17.674835 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:17.674621 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:19.674616761 +0000 UTC m=+51.192932496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : configmap "kube-rbac-proxy" not found Sep 20 01:53:17.735526 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.735499 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bq76k"] Sep 20 01:53:17.757235 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.757214 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bq76k"] Sep 20 01:53:17.757334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.757294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.759460 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.759436 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 20 01:53:17.760334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.760315 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bdvg8\"" Sep 20 01:53:17.760334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.760332 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 20 01:53:17.760498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.760333 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 20 01:53:17.760498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.760382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 20 01:53:17.876876 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.876846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-key\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.876876 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.876879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-cabundle\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.877081 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.876901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhtz\" (UniqueName: \"kubernetes.io/projected/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-kube-api-access-hdhtz\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.978272 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.978207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-key\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.978272 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.978239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-cabundle\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.978272 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.978260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhtz\" (UniqueName: \"kubernetes.io/projected/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-kube-api-access-hdhtz\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.978918 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.978877 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-cabundle\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.980724 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.980704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-signing-key\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:17.986088 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:17.986071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhtz\" (UniqueName: \"kubernetes.io/projected/a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7-kube-api-access-hdhtz\") pod \"service-ca-55f6fc7b86-bq76k\" (UID: \"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:18.065241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:18.065213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" Sep 20 01:53:18.097237 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:18.097209 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpl5v_6da7c700-b7cc-4c87-9400-ef17a3557825/dns-node-resolver/0.log" Sep 20 01:53:18.342209 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:18.342144 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bq76k"] Sep 20 01:53:18.350357 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:18.350332 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63e5f78_1ef0_4a55_a8ad_1bcfbbdcebb7.slice/crio-461d189852811baa7962da855693273e26e808cf405f9b4501a5da47aec3ba50 WatchSource:0}: Error finding container 461d189852811baa7962da855693273e26e808cf405f9b4501a5da47aec3ba50: Status 404 returned error can't find the container with id 461d189852811baa7962da855693273e26e808cf405f9b4501a5da47aec3ba50 Sep 20 01:53:18.388519 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:18.388497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" event={"ID":"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7","Type":"ContainerStarted","Data":"461d189852811baa7962da855693273e26e808cf405f9b4501a5da47aec3ba50"} Sep 20 01:53:18.897529 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:18.897502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dczpm_ee537002-c5d1-4dcf-99ae-13e1b0fe1607/node-ca/0.log" Sep 20 01:53:19.690472 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:19.690434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:19.690664 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:19.690545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:19.690664 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:19.690571 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 20 01:53:19.690769 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:19.690664 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:23.69064129 +0000 UTC m=+55.208957049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : configmap "kube-rbac-proxy" not found Sep 20 01:53:19.690769 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:19.690674 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 20 01:53:19.690769 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:19.690727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:23.690710972 +0000 UTC m=+55.209026719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : secret "insights-runtime-extractor-tls" not found Sep 20 01:53:19.897512 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:19.897488 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvbhm_2313a01c-8a54-4b2e-b113-6100216ce6b5/serve-healthcheck-canary/0.log" Sep 20 01:53:20.394406 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:20.394372 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" event={"ID":"a63e5f78-1ef0-4a55-a8ad-1bcfbbdcebb7","Type":"ContainerStarted","Data":"2b2073d33660be68ce3ba20951ea9b766c856989bd60715f6ba9a4918e2551d0"} Sep 20 01:53:20.410937 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:20.410868 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-bq76k" podStartSLOduration=1.695379159 podStartE2EDuration="3.410851643s" totalImagesPullingTime="1.715472484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:20.410318024 +0000 UTC m=+51.928633781" watchObservedRunningTime="2026-09-20 01:53:20.410851643 +0000 UTC m=+51.929167402" Sep 20 01:53:23.720350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:23.720313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:23.720933 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:23.720382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:23.720933 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:23.720439 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 20 01:53:23.720933 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:23.720475 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 20 01:53:23.720933 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:23.720519 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:31.720498169 +0000 UTC m=+63.238813920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : configmap "kube-rbac-proxy" not found Sep 20 01:53:23.720933 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:23.720538 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls podName:84ca736c-c75f-4bb0-b7ad-2dbd182acc00 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:31.720529075 +0000 UTC m=+63.238844812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-52f4v" (UID: "84ca736c-c75f-4bb0-b7ad-2dbd182acc00") : secret "insights-runtime-extractor-tls" not found Sep 20 01:53:28.337174 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:28.337146 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7b7rr" Sep 20 01:53:31.782483 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.782434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:31.782966 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.782505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:31.782966 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.782938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:31.784887 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.784866 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84ca736c-c75f-4bb0-b7ad-2dbd182acc00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52f4v\" (UID: \"84ca736c-c75f-4bb0-b7ad-2dbd182acc00\") " pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:31.870600 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.870571 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fcgnp\"" Sep 20 01:53:31.879517 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:31.879495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52f4v" Sep 20 01:53:32.148526 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:32.148494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52f4v"] Sep 20 01:53:32.151236 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:32.151205 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ca736c_c75f_4bb0_b7ad_2dbd182acc00.slice/crio-7c419f41b2ff9f50d185dc7e5650fbaa430da5d9ebc50bf62acebd5d00721311 WatchSource:0}: Error finding container 7c419f41b2ff9f50d185dc7e5650fbaa430da5d9ebc50bf62acebd5d00721311: Status 404 returned error can't find the container with id 7c419f41b2ff9f50d185dc7e5650fbaa430da5d9ebc50bf62acebd5d00721311 Sep 20 01:53:32.424749 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:32.424671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52f4v" event={"ID":"84ca736c-c75f-4bb0-b7ad-2dbd182acc00","Type":"ContainerStarted","Data":"2d740275e8694d12d489e9d11820d8176a76530b8aa6ca0f7d0e031437e4f990"} Sep 20 01:53:32.424749 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:32.424725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52f4v" event={"ID":"84ca736c-c75f-4bb0-b7ad-2dbd182acc00","Type":"ContainerStarted","Data":"7c419f41b2ff9f50d185dc7e5650fbaa430da5d9ebc50bf62acebd5d00721311"} Sep 20 01:53:33.428359 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.428323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52f4v" event={"ID":"84ca736c-c75f-4bb0-b7ad-2dbd182acc00","Type":"ContainerStarted","Data":"36e0c382d737ee083a1d93b1cb341a4f138bb39b91f33e9700af6fc42df0b967"} Sep 20 01:53:33.699094 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.699065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:33.699241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.699113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:33.701411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.701380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/420afa4e-3ba4-442c-8548-70cecec2bc11-metrics-tls\") pod \"dns-default-wdsjv\" (UID: \"420afa4e-3ba4-442c-8548-70cecec2bc11\") " pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:33.701515 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.701496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"image-registry-5fbdb58748-hnvmj\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:33.712507 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.712490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp8gx\"" Sep 20 01:53:33.720435 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.720420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:33.725971 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.725952 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9d2x\"" Sep 20 01:53:33.730737 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.730722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:33.799977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.799600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:33.799977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.799716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:33.802476 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.801980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 20 01:53:33.802476 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.802319 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 20 01:53:33.812465 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.812440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe772d7-ed21-4135-ac2d-a2922c120101-original-pull-secret\") pod \"global-pull-secret-syncer-5j9ht\" (UID: \"cfe772d7-ed21-4135-ac2d-a2922c120101\") " pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:33.814182 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.814145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e1da5e-94ec-46be-85e8-dc0ced0ff4a4-metrics-certs\") pod \"network-metrics-daemon-xwfcm\" (UID: \"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4\") " pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:33.820697 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.820344 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjlfs\"" Sep 20 01:53:33.823394 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.823342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5j9ht" Sep 20 01:53:33.828759 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.828740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwfcm" Sep 20 01:53:33.900754 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.900486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:33.902975 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.902958 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 20 01:53:33.913730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.913706 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 20 01:53:33.924866 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:33.924841 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgc8\" (UniqueName: \"kubernetes.io/projected/5ddc7b08-635c-421f-856b-93f530c903b9-kube-api-access-wlgc8\") pod \"network-check-target-pp4rc\" (UID: \"5ddc7b08-635c-421f-856b-93f530c903b9\") " pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:34.017404 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.017282 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wdsjv"] Sep 20 01:53:34.020242 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:34.020209 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420afa4e_3ba4_442c_8548_70cecec2bc11.slice/crio-1df9607f7eefb4c3340ec1ef1cd3a2c299858025663bf6b4ca397f864db07e01 WatchSource:0}: Error finding container 1df9607f7eefb4c3340ec1ef1cd3a2c299858025663bf6b4ca397f864db07e01: Status 404 returned error can't find the container with id 1df9607f7eefb4c3340ec1ef1cd3a2c299858025663bf6b4ca397f864db07e01 Sep 20 01:53:34.040797 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.040763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:53:34.045263 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:34.045240 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1975626e_8e1c_4dbc_a3fb_a5eefc3805d2.slice/crio-4c0d090831ff2ca5ff77fdf66d78ee6b4df2d8aa3dbb12a2d90286990cfcd408 WatchSource:0}: Error finding container 4c0d090831ff2ca5ff77fdf66d78ee6b4df2d8aa3dbb12a2d90286990cfcd408: Status 404 returned error can't find the container with id 4c0d090831ff2ca5ff77fdf66d78ee6b4df2d8aa3dbb12a2d90286990cfcd408 Sep 20 01:53:34.105065 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.105038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwfcm"] Sep 20 01:53:34.107598 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.107578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5j9ht"] Sep 20 01:53:34.109772 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:34.109750 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e1da5e_94ec_46be_85e8_dc0ced0ff4a4.slice/crio-6812b5fca2a08b13814ab5e2484dc09cd636a68f6f8a95477acba85e31ed954b WatchSource:0}: Error finding container 6812b5fca2a08b13814ab5e2484dc09cd636a68f6f8a95477acba85e31ed954b: Status 404 returned error can't find the container with id 6812b5fca2a08b13814ab5e2484dc09cd636a68f6f8a95477acba85e31ed954b Sep 20 01:53:34.122233 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:34.122200 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe772d7_ed21_4135_ac2d_a2922c120101.slice/crio-42dd71b75a36491a822d7242de63ed5ae705ee4ef5c0b66c3404a00114cf6fd3 WatchSource:0}: Error finding container 42dd71b75a36491a822d7242de63ed5ae705ee4ef5c0b66c3404a00114cf6fd3: Status 404 returned error can't find the container with id 42dd71b75a36491a822d7242de63ed5ae705ee4ef5c0b66c3404a00114cf6fd3 Sep 20 01:53:34.129597 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.129489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cvrtw\"" Sep 20 01:53:34.138170 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.138140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:34.415401 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.415371 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp4rc"] Sep 20 01:53:34.433858 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.433523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5j9ht" event={"ID":"cfe772d7-ed21-4135-ac2d-a2922c120101","Type":"ContainerStarted","Data":"42dd71b75a36491a822d7242de63ed5ae705ee4ef5c0b66c3404a00114cf6fd3"} Sep 20 01:53:34.435794 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.435676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" event={"ID":"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2","Type":"ContainerStarted","Data":"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd"} Sep 20 01:53:34.435794 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.435724 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" event={"ID":"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2","Type":"ContainerStarted","Data":"4c0d090831ff2ca5ff77fdf66d78ee6b4df2d8aa3dbb12a2d90286990cfcd408"} Sep 20 01:53:34.435794 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.435767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:34.437089 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.437055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwfcm" event={"ID":"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4","Type":"ContainerStarted","Data":"6812b5fca2a08b13814ab5e2484dc09cd636a68f6f8a95477acba85e31ed954b"} Sep 20 01:53:34.438378 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.438358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdsjv" event={"ID":"420afa4e-3ba4-442c-8548-70cecec2bc11","Type":"ContainerStarted","Data":"1df9607f7eefb4c3340ec1ef1cd3a2c299858025663bf6b4ca397f864db07e01"} Sep 20 01:53:34.454977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:34.454923 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" podStartSLOduration=45.454885065 podStartE2EDuration="45.454885065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:34.453702657 +0000 UTC m=+65.972018415" watchObservedRunningTime="2026-09-20 01:53:34.454885065 +0000 UTC m=+65.973200822" Sep 20 01:53:34.653578 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:34.653539 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddc7b08_635c_421f_856b_93f530c903b9.slice/crio-61b6e3a0e8187c2463961087a72bf86f511474418f37fe6a02cc578a0b694e96 WatchSource:0}: Error finding container 61b6e3a0e8187c2463961087a72bf86f511474418f37fe6a02cc578a0b694e96: Status 404 returned error can't find the container with id 61b6e3a0e8187c2463961087a72bf86f511474418f37fe6a02cc578a0b694e96 Sep 20 01:53:35.446573 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:35.446527 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52f4v" event={"ID":"84ca736c-c75f-4bb0-b7ad-2dbd182acc00","Type":"ContainerStarted","Data":"553f142f0c1436c1f5e59b19487635fca9027ee16c1cf974260a360da239d98a"} Sep 20 01:53:35.448278 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:35.448246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp4rc" event={"ID":"5ddc7b08-635c-421f-856b-93f530c903b9","Type":"ContainerStarted","Data":"61b6e3a0e8187c2463961087a72bf86f511474418f37fe6a02cc578a0b694e96"} Sep 20 01:53:35.464036 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:35.463417 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-52f4v" podStartSLOduration=18.110626223 podStartE2EDuration="20.463398714s" totalImagesPullingTime="2.352772491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:53:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:35.462183325 +0000 UTC m=+66.980499084" watchObservedRunningTime="2026-09-20 01:53:35.463398714 +0000 UTC m=+66.981714470" Sep 20 01:53:36.457780 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.457744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdsjv" event={"ID":"420afa4e-3ba4-442c-8548-70cecec2bc11","Type":"ContainerStarted","Data":"6b33127b9acc7a654ad6d68da6b416ec730fd0bafa7d86709d2d1aea172f6447"} Sep 20 01:53:36.459228 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.457838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wdsjv" event={"ID":"420afa4e-3ba4-442c-8548-70cecec2bc11","Type":"ContainerStarted","Data":"1d03cb387a514fb4568cc0ae6a89ad4ad27973e0cd1f709934146f7fcdd2faa7"} Sep 20 01:53:36.459228 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.457958 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:36.461086 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.461065 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwfcm" event={"ID":"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4","Type":"ContainerStarted","Data":"319c9de2edccc93799d8f819c8640c1407b29c0515c6fc65695f32c5ce30bb81"} Sep 20 01:53:36.461134 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.461113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwfcm" event={"ID":"70e1da5e-94ec-46be-85e8-dc0ced0ff4a4","Type":"ContainerStarted","Data":"aa5a2c3bca15c70f0da6afc2d7a4a1b0cfcc9b1858ae6e5fdd89734b5b1c3c8b"} Sep 20 01:53:36.478614 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.477940 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wdsjv" podStartSLOduration=33.299302784 podStartE2EDuration="35.477898941s" totalImagesPullingTime="2.178596157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:36.477285538 +0000 UTC m=+67.995601296" watchObservedRunningTime="2026-09-20 01:53:36.477898941 +0000 UTC m=+67.996214699" Sep 20 01:53:36.495719 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:36.495591 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwfcm" podStartSLOduration=66.136880391 podStartE2EDuration="1m7.495571836s" totalImagesPullingTime="1.358691445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:36.493952908 +0000 UTC m=+68.012268667" watchObservedRunningTime="2026-09-20 01:53:36.495571836 +0000 UTC m=+68.013887594" Sep 20 01:53:40.483939 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.483885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5j9ht" event={"ID":"cfe772d7-ed21-4135-ac2d-a2922c120101","Type":"ContainerStarted","Data":"7de650275da1218dcd2ad012773479f9400e750b83099a4c1ca547ce8878b0d8"} Sep 20 01:53:40.485465 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.485436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp4rc" event={"ID":"5ddc7b08-635c-421f-856b-93f530c903b9","Type":"ContainerStarted","Data":"ecb933bac1f1324e4e7d9c56129ba2537c8aee836a50688bd7bddec24e3dc2bc"} Sep 20 01:53:40.485586 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.485496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:53:40.497491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.497446 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5j9ht" podStartSLOduration=66.202943056 podStartE2EDuration="1m11.497433486s" totalImagesPullingTime="5.29449043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:40.497298185 +0000 UTC m=+72.015613943" watchObservedRunningTime="2026-09-20 01:53:40.497433486 +0000 UTC m=+72.015749242" Sep 20 01:53:40.510293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.510250 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pp4rc" podStartSLOduration=66.751467172 podStartE2EDuration="1m11.510236346s" totalImagesPullingTime="4.758769174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:40.509876054 +0000 UTC m=+72.028191802" watchObservedRunningTime="2026-09-20 01:53:40.510236346 +0000 UTC m=+72.028552104" Sep 20 01:53:40.999453 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:40.999423 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:53:45.065188 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.065156 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj"] Sep 20 01:53:45.068317 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.068301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.071004 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.070976 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 20 01:53:45.071270 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.071252 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 20 01:53:45.071396 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.071378 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 20 01:53:45.081040 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.081024 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7pss7\"" Sep 20 01:53:45.081167 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.081151 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 20 01:53:45.081390 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.081372 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 20 01:53:45.096052 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.096032 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj"] Sep 20 01:53:45.185522 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.185493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.185691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.185527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqgp\" (UniqueName: \"kubernetes.io/projected/1ad72dcc-eb10-465b-9944-055f3e79c1f8-kube-api-access-lcqgp\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.185691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.185563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.185691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.185586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad72dcc-eb10-465b-9944-055f3e79c1f8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.285862 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.285828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.285862 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.285860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqgp\" (UniqueName: \"kubernetes.io/projected/1ad72dcc-eb10-465b-9944-055f3e79c1f8-kube-api-access-lcqgp\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.286091 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.285886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.286091 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.285922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad72dcc-eb10-465b-9944-055f3e79c1f8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.286091 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:45.286025 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 20 01:53:45.286208 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:45.286102 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls podName:1ad72dcc-eb10-465b-9944-055f3e79c1f8 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:45.786078906 +0000 UTC m=+77.304394647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-6dfdj" (UID: "1ad72dcc-eb10-465b-9944-055f3e79c1f8") : secret "prometheus-operator-tls" not found Sep 20 01:53:45.286643 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.286570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad72dcc-eb10-465b-9944-055f3e79c1f8-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.288420 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.288400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.294487 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.294468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqgp\" (UniqueName: \"kubernetes.io/projected/1ad72dcc-eb10-465b-9944-055f3e79c1f8-kube-api-access-lcqgp\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.789055 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.789025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.791416 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.791389 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad72dcc-eb10-465b-9944-055f3e79c1f8-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-6dfdj\" (UID: \"1ad72dcc-eb10-465b-9944-055f3e79c1f8\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:45.977044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:45.977000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" Sep 20 01:53:46.244763 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:46.244733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj"] Sep 20 01:53:46.248042 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:46.248009 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad72dcc_eb10_465b_9944_055f3e79c1f8.slice/crio-97193e1dc81b344310e164b09cbc841ee71718ce460b7814c5102369155e7839 WatchSource:0}: Error finding container 97193e1dc81b344310e164b09cbc841ee71718ce460b7814c5102369155e7839: Status 404 returned error can't find the container with id 97193e1dc81b344310e164b09cbc841ee71718ce460b7814c5102369155e7839 Sep 20 01:53:46.466594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:46.466570 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wdsjv" Sep 20 01:53:46.506943 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:46.506860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" event={"ID":"1ad72dcc-eb10-465b-9944-055f3e79c1f8","Type":"ContainerStarted","Data":"97193e1dc81b344310e164b09cbc841ee71718ce460b7814c5102369155e7839"} Sep 20 01:53:47.512717 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:47.512680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" event={"ID":"1ad72dcc-eb10-465b-9944-055f3e79c1f8","Type":"ContainerStarted","Data":"905e8e415a91c3a9673ffc46669002f97754c0276c6d7b5a8b5851806456a7cf"} Sep 20 01:53:47.513061 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:47.512743 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" event={"ID":"1ad72dcc-eb10-465b-9944-055f3e79c1f8","Type":"ContainerStarted","Data":"5126dd6fe238ef9553d2ab12fd6a79ebb8d50fec65614620dca5a8b531b0a8c1"} Sep 20 01:53:47.527462 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:47.527211 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-6dfdj" podStartSLOduration=1.3768267459999999 podStartE2EDuration="2.527191473s" totalImagesPullingTime="1.150364727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:47.526645062 +0000 UTC m=+79.044960820" watchObservedRunningTime="2026-09-20 01:53:47.527191473 +0000 UTC m=+79.045507231" Sep 20 01:53:48.457054 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.457022 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:53:48.459816 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.459802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.462882 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.462866 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 20 01:53:48.463008 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.462882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hhjxq\"" Sep 20 01:53:48.463077 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 20 01:53:48.463077 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 20 01:53:48.463077 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463039 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 20 01:53:48.463294 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463278 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 20 01:53:48.463384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463370 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 20 01:53:48.463778 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.463763 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 20 01:53:48.467408 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.467390 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 20 01:53:48.469875 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.469857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:53:48.506723 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506723 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x79x\" (UniqueName: \"kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506861 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506861 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506861 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.506977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.506932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.607797 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.607766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.607831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.607858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.607879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x79x\" (UniqueName: \"kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.607925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.608081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608206 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.608131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.608949 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.608873 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.610890 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.609261 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.610890 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.609569 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.612385 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.612354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.612712 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.612682 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.614644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.614624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.616163 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.616148 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x79x\" (UniqueName: \"kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x\") pod \"console-6c97cc7665-xknnf\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:48.769092 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:48.769015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:49.037710 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.037642 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:53:49.040784 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:49.040758 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d45c59_041f_461c_a8ea_05b4a859a944.slice/crio-26701b5efb25e5f40538d8abbab78e3c4899254c05a63d23c0ac1b20857885dc WatchSource:0}: Error finding container 26701b5efb25e5f40538d8abbab78e3c4899254c05a63d23c0ac1b20857885dc: Status 404 returned error can't find the container with id 26701b5efb25e5f40538d8abbab78e3c4899254c05a63d23c0ac1b20857885dc Sep 20 01:53:49.408365 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.408335 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk"] Sep 20 01:53:49.412755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.412737 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.414975 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.414954 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 20 01:53:49.415105 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.415082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-j49mm\"" Sep 20 01:53:49.415211 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.415132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 20 01:53:49.422806 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.422786 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk"] Sep 20 01:53:49.429881 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.429861 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z9wp6"] Sep 20 01:53:49.432772 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.432757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.434961 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.434939 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 20 01:53:49.435083 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.435059 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z8bph\"" Sep 20 01:53:49.435166 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.435080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 20 01:53:49.435166 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.435112 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 20 01:53:49.441974 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.441855 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vkwfk"] Sep 20 01:53:49.444691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.444675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.447028 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.446791 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bclt5\"" Sep 20 01:53:49.447180 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.447155 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 20 01:53:49.447255 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.447196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 20 01:53:49.447255 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.447205 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 20 01:53:49.447758 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.447735 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z9wp6"] Sep 20 01:53:49.514500 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514620 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-root\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514620 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514620 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514722 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-sys\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514722 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvh4\" (UniqueName: \"kubernetes.io/projected/fe8a427a-0425-4b16-a503-5056dea13f18-kube-api-access-qfvh4\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514722 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zb7x\" (UniqueName: \"kubernetes.io/projected/88933f47-3e2c-4e34-8850-8f529e247cb6-kube-api-access-7zb7x\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.514722 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncv6\" (UniqueName: \"kubernetes.io/projected/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-kube-api-access-4ncv6\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514722 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-metrics-client-ca\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-tls\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-wtmp\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe8a427a-0425-4b16-a503-5056dea13f18-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f47-3e2c-4e34-8850-8f529e247cb6-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-textfile\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.514978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-accelerators-collector-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.514998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.515000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.515292 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.515040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.520537 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.520499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97cc7665-xknnf" event={"ID":"22d45c59-041f-461c-a8ea-05b4a859a944","Type":"ContainerStarted","Data":"26701b5efb25e5f40538d8abbab78e3c4899254c05a63d23c0ac1b20857885dc"} Sep 20 01:53:49.615644 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-sys\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvh4\" (UniqueName: \"kubernetes.io/projected/fe8a427a-0425-4b16-a503-5056dea13f18-kube-api-access-qfvh4\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zb7x\" (UniqueName: \"kubernetes.io/projected/88933f47-3e2c-4e34-8850-8f529e247cb6-kube-api-access-7zb7x\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncv6\" (UniqueName: \"kubernetes.io/projected/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-kube-api-access-4ncv6\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-metrics-client-ca\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-tls\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.615854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-wtmp\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616035 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-wtmp\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-sys\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe8a427a-0425-4b16-a503-5056dea13f18-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f47-3e2c-4e34-8850-8f529e247cb6-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-textfile\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-accelerators-collector-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.616538 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-root\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.617121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-root\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.617121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-metrics-client-ca\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.617121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.616782 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-textfile\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.617121 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:49.616812 2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 20 01:53:49.617121 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:53:49.616890 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls podName:88933f47-3e2c-4e34-8850-8f529e247cb6 nodeName:}" failed. No retries permitted until 2026-09-20 01:53:50.116868043 +0000 UTC m=+81.635183795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-8s2nk" (UID: "88933f47-3e2c-4e34-8850-8f529e247cb6") : secret "openshift-state-metrics-tls" not found Sep 20 01:53:49.617363 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.617119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.617363 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.617248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-accelerators-collector-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.617470 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.617388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe8a427a-0425-4b16-a503-5056dea13f18-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.617695 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.617670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8a427a-0425-4b16-a503-5056dea13f18-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.617958 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.617933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88933f47-3e2c-4e34-8850-8f529e247cb6-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.619042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.618816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.619042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.618990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe8a427a-0425-4b16-a503-5056dea13f18-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.620117 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.620094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.620600 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.620581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.620739 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.620661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-node-exporter-tls\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.623814 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.623795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvh4\" (UniqueName: \"kubernetes.io/projected/fe8a427a-0425-4b16-a503-5056dea13f18-kube-api-access-qfvh4\") pod \"kube-state-metrics-84c885658c-z9wp6\" (UID: \"fe8a427a-0425-4b16-a503-5056dea13f18\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.624254 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.624227 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zb7x\" (UniqueName: \"kubernetes.io/projected/88933f47-3e2c-4e34-8850-8f529e247cb6-kube-api-access-7zb7x\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:49.625507 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.625488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncv6\" (UniqueName: \"kubernetes.io/projected/0afd4755-1f3e-41c3-ab46-98d95e2ffb23-kube-api-access-4ncv6\") pod \"node-exporter-vkwfk\" (UID: \"0afd4755-1f3e-41c3-ab46-98d95e2ffb23\") " pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:49.741826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.741323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" Sep 20 01:53:49.754585 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:49.754564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vkwfk" Sep 20 01:53:50.045881 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.045671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-z9wp6"] Sep 20 01:53:50.052340 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:50.052307 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8a427a_0425_4b16_a503_5056dea13f18.slice/crio-e104ebcfd0762a5b5a2b825b7a19ea1b6032b6d59e22715f4791791fed5f6260 WatchSource:0}: Error finding container e104ebcfd0762a5b5a2b825b7a19ea1b6032b6d59e22715f4791791fed5f6260: Status 404 returned error can't find the container with id e104ebcfd0762a5b5a2b825b7a19ea1b6032b6d59e22715f4791791fed5f6260 Sep 20 01:53:50.121919 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.121285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:50.125016 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.124991 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88933f47-3e2c-4e34-8850-8f529e247cb6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-8s2nk\" (UID: \"88933f47-3e2c-4e34-8850-8f529e247cb6\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:50.321977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.321622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" Sep 20 01:53:50.534849 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.531342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vkwfk" event={"ID":"0afd4755-1f3e-41c3-ab46-98d95e2ffb23","Type":"ContainerStarted","Data":"7dc7352a38c91edbae8767b423c56a144471d132e3365a35592c00a7c32881fd"} Sep 20 01:53:50.536149 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.536058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" event={"ID":"fe8a427a-0425-4b16-a503-5056dea13f18","Type":"ContainerStarted","Data":"e104ebcfd0762a5b5a2b825b7a19ea1b6032b6d59e22715f4791791fed5f6260"} Sep 20 01:53:50.623442 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:50.623367 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk"] Sep 20 01:53:50.627031 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:50.627003 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88933f47_3e2c_4e34_8850_8f529e247cb6.slice/crio-a073f7519a4cb418260f011a8483ad0ab2f8b4fb462521e309d9148e78574cd7 WatchSource:0}: Error finding container a073f7519a4cb418260f011a8483ad0ab2f8b4fb462521e309d9148e78574cd7: Status 404 returned error can't find the container with id a073f7519a4cb418260f011a8483ad0ab2f8b4fb462521e309d9148e78574cd7 Sep 20 01:53:51.007332 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.007303 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:53:51.398687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.398654 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65d66cddf-7hhml"] Sep 20 01:53:51.402180 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.402156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.404325 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404269 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 20 01:53:51.404325 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404286 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 20 01:53:51.404325 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404269 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 20 01:53:51.404553 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404368 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 20 01:53:51.404553 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 20 01:53:51.404877 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404857 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pmdvf\"" Sep 20 01:53:51.404977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.404890 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7mcvchht9g2u1\"" Sep 20 01:53:51.415169 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.415149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65d66cddf-7hhml"] Sep 20 01:53:51.534073 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-grpc-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-metrics-client-ca\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wdn\" (UniqueName: \"kubernetes.io/projected/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-kube-api-access-n8wdn\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.534410 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.534343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.541474 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.541446 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" event={"ID":"88933f47-3e2c-4e34-8850-8f529e247cb6","Type":"ContainerStarted","Data":"a073f7519a4cb418260f011a8483ad0ab2f8b4fb462521e309d9148e78574cd7"} Sep 20 01:53:51.635490 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-metrics-client-ca\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.635490 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wdn\" (UniqueName: \"kubernetes.io/projected/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-kube-api-access-n8wdn\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-grpc-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.635741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.636741 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.636680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-metrics-client-ca\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.638734 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.638690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.638888 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.638868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-grpc-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.638888 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.638879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.639042 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.639015 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.639162 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.639138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.640465 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.640446 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-secret-thanos-querier-tls\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.643609 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.643588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wdn\" (UniqueName: \"kubernetes.io/projected/5ab2da1e-db3a-48ad-a8da-ffdb3ac44875-kube-api-access-n8wdn\") pod \"thanos-querier-65d66cddf-7hhml\" (UID: \"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875\") " pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:51.713072 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:51.713015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:54.155830 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.155797 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65d66cddf-7hhml"] Sep 20 01:53:54.167036 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:54.166766 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab2da1e_db3a_48ad_a8da_ffdb3ac44875.slice/crio-a2c1c228b123c9477673aa17eb8890d10b9e457b9ac1c84c5ed1b5d663210a23 WatchSource:0}: Error finding container a2c1c228b123c9477673aa17eb8890d10b9e457b9ac1c84c5ed1b5d663210a23: Status 404 returned error can't find the container with id a2c1c228b123c9477673aa17eb8890d10b9e457b9ac1c84c5ed1b5d663210a23 Sep 20 01:53:54.240969 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.240935 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:53:54.244845 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.244824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.254565 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.254536 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:53:54.360877 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.360841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361052 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.360894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361052 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.360948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361052 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.361009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361052 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.361045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361197 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.361068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frc46\" (UniqueName: \"kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.361197 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.361135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462350 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frc46\" (UniqueName: \"kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.462621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.462456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.463157 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.463124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.463425 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.463276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.463425 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.463313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.463425 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.463320 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.465092 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.465068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.465496 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.465480 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.470716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.470692 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frc46\" (UniqueName: \"kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46\") pod \"console-7945fc49b9-c7zjm\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.554960 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.554918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97cc7665-xknnf" event={"ID":"22d45c59-041f-461c-a8ea-05b4a859a944","Type":"ContainerStarted","Data":"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4"} Sep 20 01:53:54.555204 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.555187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:53:54.556777 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.556755 2578 generic.go:356] "Generic (PLEG): container finished" podID="0afd4755-1f3e-41c3-ab46-98d95e2ffb23" containerID="1e3d4e00e309a0801f02fae3e71b244bf86998ac485a10290bdcdddbda227fe7" exitCode=0 Sep 20 01:53:54.556891 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.556825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vkwfk" event={"ID":"0afd4755-1f3e-41c3-ab46-98d95e2ffb23","Type":"ContainerDied","Data":"1e3d4e00e309a0801f02fae3e71b244bf86998ac485a10290bdcdddbda227fe7"} Sep 20 01:53:54.558258 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.558236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"a2c1c228b123c9477673aa17eb8890d10b9e457b9ac1c84c5ed1b5d663210a23"} Sep 20 01:53:54.560643 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.560618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" event={"ID":"fe8a427a-0425-4b16-a503-5056dea13f18","Type":"ContainerStarted","Data":"b56c0ba1cb067665c18da2dc7add1a8f5405f9f53a9b4273c27efcb71b65f2cd"} Sep 20 01:53:54.560810 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.560666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" event={"ID":"fe8a427a-0425-4b16-a503-5056dea13f18","Type":"ContainerStarted","Data":"1d04b05c8cb64e43d8b7ee55ff8517f54c102a41bf229f172b5a1e11e7cf6f72"} Sep 20 01:53:54.560810 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.560697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" event={"ID":"fe8a427a-0425-4b16-a503-5056dea13f18","Type":"ContainerStarted","Data":"53d73c06236399eacc64754dc69ed0a7eceaeff25bd4aadaf25b4386e80b1e76"} Sep 20 01:53:54.562531 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.562511 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" event={"ID":"88933f47-3e2c-4e34-8850-8f529e247cb6","Type":"ContainerStarted","Data":"f155dc39eeb8938b41f6188132790f2de2847d7cb3f52f7a804d7f4aa0b3069f"} Sep 20 01:53:54.562623 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.562557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" event={"ID":"88933f47-3e2c-4e34-8850-8f529e247cb6","Type":"ContainerStarted","Data":"cb4865b4a1c6e5ce53617a9e804133fd43c7171d84c000899245b097214ce459"} Sep 20 01:53:54.571236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.571156 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c97cc7665-xknnf" podStartSLOduration=1.673128543 podStartE2EDuration="6.571139876s" totalImagesPullingTime="4.898011333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:54.570799085 +0000 UTC m=+86.089114844" watchObservedRunningTime="2026-09-20 01:53:54.571139876 +0000 UTC m=+86.089455634" Sep 20 01:53:54.610360 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.610274 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-z9wp6" podStartSLOduration=1.791274076 podStartE2EDuration="5.610250584s" totalImagesPullingTime="3.818976508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:54.609151157 +0000 UTC m=+86.127466914" watchObservedRunningTime="2026-09-20 01:53:54.610250584 +0000 UTC m=+86.128566343" Sep 20 01:53:54.829972 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:54.829868 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:53:54.833672 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:53:54.833631 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb914fef_6c00_4c6a_b8e2_ace31f0b535b.slice/crio-eab4bf622bb50ac17242e2485f9c25594445a70e9d8c1dac2ee22b5d293f50cb WatchSource:0}: Error finding container eab4bf622bb50ac17242e2485f9c25594445a70e9d8c1dac2ee22b5d293f50cb: Status 404 returned error can't find the container with id eab4bf622bb50ac17242e2485f9c25594445a70e9d8c1dac2ee22b5d293f50cb Sep 20 01:53:55.569812 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.569772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945fc49b9-c7zjm" event={"ID":"db914fef-6c00-4c6a-b8e2-ace31f0b535b","Type":"ContainerStarted","Data":"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7"} Sep 20 01:53:55.570336 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.569835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945fc49b9-c7zjm" event={"ID":"db914fef-6c00-4c6a-b8e2-ace31f0b535b","Type":"ContainerStarted","Data":"eab4bf622bb50ac17242e2485f9c25594445a70e9d8c1dac2ee22b5d293f50cb"} Sep 20 01:53:55.579070 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.579038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" event={"ID":"88933f47-3e2c-4e34-8850-8f529e247cb6","Type":"ContainerStarted","Data":"394b4d19d3896f8296bf66429ec3209c4f13b54534e8cd5ab154e20f189c0aee"} Sep 20 01:53:55.581800 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.581772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vkwfk" event={"ID":"0afd4755-1f3e-41c3-ab46-98d95e2ffb23","Type":"ContainerStarted","Data":"c9f2bd88c5d735534863a50815551f44a5ac156ae7c1a6c45dcc918fd3c2170e"} Sep 20 01:53:55.581942 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.581827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vkwfk" event={"ID":"0afd4755-1f3e-41c3-ab46-98d95e2ffb23","Type":"ContainerStarted","Data":"29a77837d66024cbbff9289b7fa90f8b9b3ea2e0578d12dc1fecde4326a956f3"} Sep 20 01:53:55.590964 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.590869 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7945fc49b9-c7zjm" podStartSLOduration=1.590853007 podStartE2EDuration="1.590853007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:55.588089208 +0000 UTC m=+87.106404967" watchObservedRunningTime="2026-09-20 01:53:55.590853007 +0000 UTC m=+87.109168768" Sep 20 01:53:55.615757 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.615700 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vkwfk" podStartSLOduration=2.424312953 podStartE2EDuration="6.615679577s" totalImagesPullingTime="4.096673174s" totalInitContainerRuntime="94.69345ms" isStatefulPod=true podCreationTimestamp="2026-09-20 01:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:55.613839323 +0000 UTC m=+87.132155163" watchObservedRunningTime="2026-09-20 01:53:55.615679577 +0000 UTC m=+87.133995334" Sep 20 01:53:55.644061 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:55.643970 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-8s2nk" podStartSLOduration=5.49928068 podStartE2EDuration="6.643949215s" totalImagesPullingTime="1.144668535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:55.643441383 +0000 UTC m=+87.161757189" watchObservedRunningTime="2026-09-20 01:53:55.643949215 +0000 UTC m=+87.162264972" Sep 20 01:53:56.588652 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:56.588557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"5018bd5959e9a7be2ef629f5356b9c32936d98655e768c94cc41662c899753a9"} Sep 20 01:53:56.588652 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:56.588638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"1538aa4cc4967d30e394d655ef02a5e78e26324f62bcc0a7f00e2803d933a50c"} Sep 20 01:53:56.589200 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:56.588679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"65a108909c1061348bc43f4e81ca163868ab9e7d216173d03f6aa6edbc447e85"} Sep 20 01:53:57.595733 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:57.595708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"66d43cabfbfab2da5ff1d258e5a35899afb7358f0de6e90a48061f6993345aef"} Sep 20 01:53:57.595997 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:57.595773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"2488187ca9cde047659dd233d6c7325a86d46a4f0f40fe422fbe2e9c79b49b69"} Sep 20 01:53:58.602802 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.602764 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" event={"ID":"5ab2da1e-db3a-48ad-a8da-ffdb3ac44875","Type":"ContainerStarted","Data":"8698a4252431046a83ac92d47431c3997c0163bfdb540309b359995a55ea3895"} Sep 20 01:53:58.603172 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.602920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:53:58.625891 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.625849 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" podStartSLOduration=4.506074359 podStartE2EDuration="7.625836062s" totalImagesPullingTime="3.119761703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:53:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:53:58.625514632 +0000 UTC m=+90.143830388" watchObservedRunningTime="2026-09-20 01:53:58.625836062 +0000 UTC m=+90.144151819" Sep 20 01:53:58.770117 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.770079 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:58.770282 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.770179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:58.774741 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:58.774722 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:53:59.610026 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:53:59.610000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:54:04.555854 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.555821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:54:04.556282 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.555874 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:54:04.560315 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.560292 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:54:04.612325 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.612302 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65d66cddf-7hhml" Sep 20 01:54:04.628167 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.628145 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:54:04.689086 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:04.689056 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:54:06.018529 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.018484 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" podUID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" containerName="registry" containerID="cri-o://ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd" gracePeriod=30 Sep 20 01:54:06.284599 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.284573 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:54:06.361756 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361687 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361756 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361724 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361761 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361784 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361860 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361897 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd9d6\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.361981 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.361954 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca\") pod \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\" (UID: \"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2\") " Sep 20 01:54:06.362286 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.362209 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:06.362517 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.362484 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:06.364739 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.364711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:54:06.365070 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.365045 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:54:06.365160 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.365113 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:54:06.365160 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.365128 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:54:06.365256 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.365203 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6" (OuterVolumeSpecName: "kube-api-access-vd9d6") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "kube-api-access-vd9d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:54:06.370279 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.370253 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" (UID: "1975626e-8e1c-4dbc-a3fb-a5eefc3805d2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:54:06.462517 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462490 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-certificates\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462517 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462513 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-bound-sa-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462526 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-image-registry-private-configuration\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462535 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-installation-pull-secrets\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462545 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-registry-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462552 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9d6\" (UniqueName: \"kubernetes.io/projected/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-kube-api-access-vd9d6\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462559 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-trusted-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.462716 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.462567 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2-ca-trust-extracted\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:06.634678 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.634577 2578 generic.go:356] "Generic (PLEG): container finished" podID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" containerID="ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd" exitCode=0 Sep 20 01:54:06.634678 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.634621 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" Sep 20 01:54:06.634678 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.634666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" event={"ID":"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2","Type":"ContainerDied","Data":"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd"} Sep 20 01:54:06.634883 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.634744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbdb58748-hnvmj" event={"ID":"1975626e-8e1c-4dbc-a3fb-a5eefc3805d2","Type":"ContainerDied","Data":"4c0d090831ff2ca5ff77fdf66d78ee6b4df2d8aa3dbb12a2d90286990cfcd408"} Sep 20 01:54:06.634883 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.634766 2578 scope.go:117] "RemoveContainer" containerID="ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd" Sep 20 01:54:06.645746 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.645728 2578 scope.go:117] "RemoveContainer" containerID="ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd" Sep 20 01:54:06.646172 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:54:06.646144 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd\": container with ID starting with ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd not found: ID does not exist" containerID="ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd" Sep 20 01:54:06.646245 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.646184 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd"} err="failed to get container status \"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd\": rpc error: code = NotFound desc = could not find container \"ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd\": container with ID starting with ea914de0b202315a7e023146ca16d20aafa4f41856389d099fdc0a2d93080cfd not found: ID does not exist" Sep 20 01:54:06.655205 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.655184 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:54:06.661059 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:06.661012 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fbdb58748-hnvmj"] Sep 20 01:54:07.013717 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:07.013678 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" path="/var/lib/kubelet/pods/1975626e-8e1c-4dbc-a3fb-a5eefc3805d2/volumes" Sep 20 01:54:11.491886 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:11.491855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pp4rc" Sep 20 01:54:29.715169 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:29.715102 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c97cc7665-xknnf" podUID="22d45c59-041f-461c-a8ea-05b4a859a944" containerName="console" containerID="cri-o://ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4" gracePeriod=15 Sep 20 01:54:30.481638 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.481611 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:54:30.544592 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544557 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544610 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x79x\" (UniqueName: \"kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544676 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544717 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.544764 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.544756 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert\") pod \"22d45c59-041f-461c-a8ea-05b4a859a944\" (UID: \"22d45c59-041f-461c-a8ea-05b4a859a944\") " Sep 20 01:54:30.545090 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.545036 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config" (OuterVolumeSpecName: "console-config") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:30.545258 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.545227 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca" (OuterVolumeSpecName: "service-ca") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:30.545334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.545236 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:30.545334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.545290 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:54:30.547192 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.547170 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:54:30.547192 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.547178 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x" (OuterVolumeSpecName: "kube-api-access-2x79x") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "kube-api-access-2x79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:54:30.547288 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.547240 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "22d45c59-041f-461c-a8ea-05b4a859a944" (UID: "22d45c59-041f-461c-a8ea-05b4a859a944"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:54:30.646221 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646192 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-console-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646221 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646215 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x79x\" (UniqueName: \"kubernetes.io/projected/22d45c59-041f-461c-a8ea-05b4a859a944-kube-api-access-2x79x\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646226 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646235 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-trusted-ca-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646242 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-oauth-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646250 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22d45c59-041f-461c-a8ea-05b4a859a944-console-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.646384 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.646257 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22d45c59-041f-461c-a8ea-05b4a859a944-oauth-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:54:30.721498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.721465 2578 generic.go:356] "Generic (PLEG): container finished" podID="22d45c59-041f-461c-a8ea-05b4a859a944" containerID="ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4" exitCode=0 Sep 20 01:54:30.721813 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.721510 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97cc7665-xknnf" Sep 20 01:54:30.721813 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.721550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97cc7665-xknnf" event={"ID":"22d45c59-041f-461c-a8ea-05b4a859a944","Type":"ContainerDied","Data":"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4"} Sep 20 01:54:30.721813 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.721601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97cc7665-xknnf" event={"ID":"22d45c59-041f-461c-a8ea-05b4a859a944","Type":"ContainerDied","Data":"26701b5efb25e5f40538d8abbab78e3c4899254c05a63d23c0ac1b20857885dc"} Sep 20 01:54:30.721813 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.721621 2578 scope.go:117] "RemoveContainer" containerID="ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4" Sep 20 01:54:30.729976 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.729961 2578 scope.go:117] "RemoveContainer" containerID="ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4" Sep 20 01:54:30.730245 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:54:30.730223 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4\": container with ID starting with ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4 not found: ID does not exist" containerID="ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4" Sep 20 01:54:30.730304 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.730257 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4"} err="failed to get container status \"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4\": rpc error: code = NotFound desc = could not find container \"ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4\": container with ID starting with ec68f4e2689d80c6b8b6db1867da70d9748f1bdffe8311b5a5a15994fbd874b4 not found: ID does not exist" Sep 20 01:54:30.742900 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.742876 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:54:30.748518 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:30.748499 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c97cc7665-xknnf"] Sep 20 01:54:31.013261 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:54:31.013187 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d45c59-041f-461c-a8ea-05b4a859a944" path="/var/lib/kubelet/pods/22d45c59-041f-461c-a8ea-05b4a859a944/volumes" Sep 20 01:55:11.131970 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.131938 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132205 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d45c59-041f-461c-a8ea-05b4a859a944" containerName="console" Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132216 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d45c59-041f-461c-a8ea-05b4a859a944" containerName="console" Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132226 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" containerName="registry" Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132232 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" containerName="registry" Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132289 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d45c59-041f-461c-a8ea-05b4a859a944" containerName="console" Sep 20 01:55:11.132381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.132297 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1975626e-8e1c-4dbc-a3fb-a5eefc3805d2" containerName="registry" Sep 20 01:55:11.137180 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.137110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.140002 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.139980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 20 01:55:11.140484 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.140468 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tnh7f\"" Sep 20 01:55:11.140546 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.140524 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 20 01:55:11.141293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141274 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 20 01:55:11.141492 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141280 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 20 01:55:11.141492 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141280 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 20 01:55:11.141640 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 20 01:55:11.141640 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 20 01:55:11.141640 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.141353 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 20 01:55:11.145498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.145482 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 20 01:55:11.149455 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.149435 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 20 01:55:11.242276 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpl2h\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-kube-api-access-vpl2h\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.242826 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.242638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343481 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpl2h\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-kube-api-access-vpl2h\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.343687 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.344149 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.344149 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.343733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.344550 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.344524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.344712 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.344566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.345610 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.345588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426ace4-9114-4757-b95e-aac93c12f2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.346813 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.346789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347299 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347540 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347611 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347529 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347611 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347683 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.347795 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.347777 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426ace4-9114-4757-b95e-aac93c12f2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.348389 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.348374 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.348791 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.348774 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c426ace4-9114-4757-b95e-aac93c12f2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.353043 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.353021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpl2h\" (UniqueName: \"kubernetes.io/projected/c426ace4-9114-4757-b95e-aac93c12f2d3-kube-api-access-vpl2h\") pod \"alertmanager-main-0\" (UID: \"c426ace4-9114-4757-b95e-aac93c12f2d3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.447886 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.447861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 20 01:55:11.721780 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.721616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 20 01:55:11.724708 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:55:11.724682 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc426ace4_9114_4757_b95e_aac93c12f2d3.slice/crio-c6203255f88ebc87fabfae8beb21596e3b5a63bfebfa210b14f2edb647072baa WatchSource:0}: Error finding container c6203255f88ebc87fabfae8beb21596e3b5a63bfebfa210b14f2edb647072baa: Status 404 returned error can't find the container with id c6203255f88ebc87fabfae8beb21596e3b5a63bfebfa210b14f2edb647072baa Sep 20 01:55:11.873438 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:11.873402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"c6203255f88ebc87fabfae8beb21596e3b5a63bfebfa210b14f2edb647072baa"} Sep 20 01:55:13.695876 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.695839 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-647cd948-kwzbx"] Sep 20 01:55:13.698176 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.698155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.700411 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.700381 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 20 01:55:13.700660 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.700634 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 20 01:55:13.700790 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.700768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vsx4r\"" Sep 20 01:55:13.700864 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.700828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 20 01:55:13.700939 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.700873 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 20 01:55:13.701142 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.701126 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 20 01:55:13.705588 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.705568 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 20 01:55:13.711099 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.711078 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-647cd948-kwzbx"] Sep 20 01:55:13.762519 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762519 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-serving-certs-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknns\" (UniqueName: \"kubernetes.io/projected/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-kube-api-access-rknns\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-metrics-client-ca\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762900 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-federate-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.762900 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.762800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.863979 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.863945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.863986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknns\" (UniqueName: \"kubernetes.io/projected/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-kube-api-access-rknns\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-metrics-client-ca\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-federate-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864387 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.864602 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.864580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-serving-certs-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.865319 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.865067 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-metrics-client-ca\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.865319 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.865262 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.865545 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.865523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-serving-certs-ca-bundle\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.867029 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.867002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-telemeter-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.867133 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.867075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.867193 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.867133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-federate-client-tls\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.868607 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.868583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.872278 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.872255 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknns\" (UniqueName: \"kubernetes.io/projected/2a3a1741-cebb-4a8d-a7cb-a061d02bde83-kube-api-access-rknns\") pod \"telemeter-client-647cd948-kwzbx\" (UID: \"2a3a1741-cebb-4a8d-a7cb-a061d02bde83\") " pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:13.882426 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.882403 2578 generic.go:356] "Generic (PLEG): container finished" podID="c426ace4-9114-4757-b95e-aac93c12f2d3" containerID="52a59f89d652466c8aeca6d4ca13aceb42851722c2e1918b84b5894b3e71a2b8" exitCode=0 Sep 20 01:55:13.882520 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:13.882461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerDied","Data":"52a59f89d652466c8aeca6d4ca13aceb42851722c2e1918b84b5894b3e71a2b8"} Sep 20 01:55:14.009727 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:14.009637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" Sep 20 01:55:14.292394 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:14.292370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-647cd948-kwzbx"] Sep 20 01:55:14.294534 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:55:14.294504 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3a1741_cebb_4a8d_a7cb_a061d02bde83.slice/crio-96d436fc4fc03f0c89e8a2d3893a2ab728d72cd4ed4c8ff24cffca3c6727815a WatchSource:0}: Error finding container 96d436fc4fc03f0c89e8a2d3893a2ab728d72cd4ed4c8ff24cffca3c6727815a: Status 404 returned error can't find the container with id 96d436fc4fc03f0c89e8a2d3893a2ab728d72cd4ed4c8ff24cffca3c6727815a Sep 20 01:55:14.887627 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:14.887595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" event={"ID":"2a3a1741-cebb-4a8d-a7cb-a061d02bde83","Type":"ContainerStarted","Data":"96d436fc4fc03f0c89e8a2d3893a2ab728d72cd4ed4c8ff24cffca3c6727815a"} Sep 20 01:55:16.898894 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.898859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"9d319b48d1988d623da2ab66394fc54872c65132afe90536535658f00d3e11fb"} Sep 20 01:55:16.899352 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.898952 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"67dce2cbcdbe3ecf79bfbd88f2777f63f948daba7df164c30324bcd1f669400a"} Sep 20 01:55:16.899352 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.898991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"47d4bfdffd0b4695047bf9dd8e7df5d9b7bf03154645a31e753cc5629270c1b8"} Sep 20 01:55:16.899352 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.899029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"3af7cba72c2b43824421341a47ac44bb9cb013195d06d83bf07536892679aace"} Sep 20 01:55:16.899352 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.899068 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"41d7f15f45f1c17c4dbac836601c4e71e2e31f52097c24c55e518d6c44785a5a"} Sep 20 01:55:16.899352 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.899107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c426ace4-9114-4757-b95e-aac93c12f2d3","Type":"ContainerStarted","Data":"5c1209a6e3dbfa238be36d2d1ca9952f78df21dda7b18a07086ab4430193f546"} Sep 20 01:55:16.901039 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.901012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" event={"ID":"2a3a1741-cebb-4a8d-a7cb-a061d02bde83","Type":"ContainerStarted","Data":"6126170fa44e51aa549c63aed3c9d6a5df3dc3c0d6cff0e1df4f9e64e83b3547"} Sep 20 01:55:16.901124 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.901072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" event={"ID":"2a3a1741-cebb-4a8d-a7cb-a061d02bde83","Type":"ContainerStarted","Data":"ae77107685000b2b0f888984e1e6c68d6a8883fb8b6f43be5e5e00995d015992"} Sep 20 01:55:16.901124 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.901102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" event={"ID":"2a3a1741-cebb-4a8d-a7cb-a061d02bde83","Type":"ContainerStarted","Data":"ab81b75276dfd392314defc5f3b18add9b100c1b895de134d133657b0ca47def"} Sep 20 01:55:16.925855 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.924529 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.726524489 podStartE2EDuration="5.924499971s" totalImagesPullingTime="3.177967098s" totalInitContainerRuntime="20.008384ms" isStatefulPod=false podCreationTimestamp="2026-09-20 01:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:55:16.922822562 +0000 UTC m=+168.441138324" watchObservedRunningTime="2026-09-20 01:55:16.924499971 +0000 UTC m=+168.442815730" Sep 20 01:55:16.942765 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:16.942714 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-647cd948-kwzbx" podStartSLOduration=2.341356897 podStartE2EDuration="3.94267767s" totalImagesPullingTime="1.601320773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:55:16.942265703 +0000 UTC m=+168.460581460" watchObservedRunningTime="2026-09-20 01:55:16.94267767 +0000 UTC m=+168.460993429" Sep 20 01:55:17.019751 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.019718 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:55:17.021742 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.021724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.031415 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.031391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:55:17.091691 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xq7g\" (UniqueName: \"kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.091853 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.091836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192621 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xq7g\" (UniqueName: \"kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.192844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.192804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.193403 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.193378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.193496 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.193398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.193496 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.193410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.194203 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.194180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.195097 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.195072 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.195254 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.195238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.199567 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.199551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xq7g\" (UniqueName: \"kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g\") pod \"console-767d987f49-7pj74\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.330682 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.330642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:17.605625 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.605600 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:55:17.608086 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:55:17.608062 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc647ec3b_ad03_431d_a145_94f7f2d8193c.slice/crio-1f3a1a56e0ab9dc39fec913cacc8ac1152641a54f51cb818b3925416f73fdc67 WatchSource:0}: Error finding container 1f3a1a56e0ab9dc39fec913cacc8ac1152641a54f51cb818b3925416f73fdc67: Status 404 returned error can't find the container with id 1f3a1a56e0ab9dc39fec913cacc8ac1152641a54f51cb818b3925416f73fdc67 Sep 20 01:55:17.909281 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.909245 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767d987f49-7pj74" event={"ID":"c647ec3b-ad03-431d-a145-94f7f2d8193c","Type":"ContainerStarted","Data":"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3"} Sep 20 01:55:17.909727 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.909303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767d987f49-7pj74" event={"ID":"c647ec3b-ad03-431d-a145-94f7f2d8193c","Type":"ContainerStarted","Data":"1f3a1a56e0ab9dc39fec913cacc8ac1152641a54f51cb818b3925416f73fdc67"} Sep 20 01:55:17.926176 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:17.926127 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-767d987f49-7pj74" podStartSLOduration=0.926107035 podStartE2EDuration="926.107035ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:55:17.924373168 +0000 UTC m=+169.442688937" watchObservedRunningTime="2026-09-20 01:55:17.926107035 +0000 UTC m=+169.444422800" Sep 20 01:55:27.331774 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:27.331696 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:27.331774 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:27.331765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:27.336199 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:27.336181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:27.955505 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:27.955479 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:55:28.001819 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:28.001790 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:55:53.020786 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.020730 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7945fc49b9-c7zjm" podUID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" containerName="console" containerID="cri-o://eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7" gracePeriod=15 Sep 20 01:55:53.259273 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.259253 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:55:53.269440 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269516 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269458 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269516 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269489 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269676 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269658 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269695 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frc46\" (UniqueName: \"kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269719 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269815 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269737 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert\") pod \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\" (UID: \"db914fef-6c00-4c6a-b8e2-ace31f0b535b\") " Sep 20 01:55:53.269815 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269797 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca" (OuterVolumeSpecName: "service-ca") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:55:53.270009 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.269990 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.270094 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.270071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:55:53.270144 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.270091 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config" (OuterVolumeSpecName: "console-config") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:55:53.270187 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.270170 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:55:53.271780 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.271722 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46" (OuterVolumeSpecName: "kube-api-access-frc46") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "kube-api-access-frc46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:55:53.272180 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.272148 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:55:53.272264 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.272196 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "db914fef-6c00-4c6a-b8e2-ace31f0b535b" (UID: "db914fef-6c00-4c6a-b8e2-ace31f0b535b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:55:53.370502 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370469 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-oauth-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.370502 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370496 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.370502 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370507 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-trusted-ca-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.370724 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370518 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frc46\" (UniqueName: \"kubernetes.io/projected/db914fef-6c00-4c6a-b8e2-ace31f0b535b-kube-api-access-frc46\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.370724 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370540 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-console-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:53.370724 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:53.370551 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db914fef-6c00-4c6a-b8e2-ace31f0b535b-oauth-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:55:54.049659 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.049631 2578 generic.go:356] "Generic (PLEG): container finished" podID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" containerID="eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7" exitCode=0 Sep 20 01:55:54.049659 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.049663 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945fc49b9-c7zjm" event={"ID":"db914fef-6c00-4c6a-b8e2-ace31f0b535b","Type":"ContainerDied","Data":"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7"} Sep 20 01:55:54.050212 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.049686 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945fc49b9-c7zjm" Sep 20 01:55:54.050212 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.049701 2578 scope.go:117] "RemoveContainer" containerID="eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7" Sep 20 01:55:54.050212 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.049692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945fc49b9-c7zjm" event={"ID":"db914fef-6c00-4c6a-b8e2-ace31f0b535b","Type":"ContainerDied","Data":"eab4bf622bb50ac17242e2485f9c25594445a70e9d8c1dac2ee22b5d293f50cb"} Sep 20 01:55:54.058135 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.058119 2578 scope.go:117] "RemoveContainer" containerID="eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7" Sep 20 01:55:54.058397 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:55:54.058378 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7\": container with ID starting with eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7 not found: ID does not exist" containerID="eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7" Sep 20 01:55:54.058463 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.058404 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7"} err="failed to get container status \"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7\": rpc error: code = NotFound desc = could not find container \"eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7\": container with ID starting with eef2672cf68375014b9ced9369596a2c825a281f5e0d3ec07c147b24a25494d7 not found: ID does not exist" Sep 20 01:55:54.068850 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.068831 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:55:54.073288 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:54.073269 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7945fc49b9-c7zjm"] Sep 20 01:55:55.014139 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:55:55.014110 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" path="/var/lib/kubelet/pods/db914fef-6c00-4c6a-b8e2-ace31f0b535b/volumes" Sep 20 01:56:31.963136 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.963103 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:56:31.963730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.963405 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" containerName="console" Sep 20 01:56:31.963730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.963415 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" containerName="console" Sep 20 01:56:31.963730 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.963463 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db914fef-6c00-4c6a-b8e2-ace31f0b535b" containerName="console" Sep 20 01:56:31.967373 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.967357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:31.977713 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:31.977693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:56:32.073033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.072998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073214 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073356 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073433 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073549 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.073549 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.073534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkf4t\" (UniqueName: \"kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174046 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174046 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174236 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174304 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174304 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174304 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkf4t\" (UniqueName: \"kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.174871 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.175009 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.175009 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.174924 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.175253 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.175231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.177189 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.177166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.177253 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.177208 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.181695 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.181679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkf4t\" (UniqueName: \"kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t\") pod \"console-74b9558bfb-l72p4\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.275636 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.275546 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:32.547211 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:32.547140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:56:32.550047 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:56:32.550021 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92be662_6898_4c73_af02_c415ad35bcf1.slice/crio-3eb56983d50581d2149436e92e873886e306412c9f1b9b8d76263ca042acf12f WatchSource:0}: Error finding container 3eb56983d50581d2149436e92e873886e306412c9f1b9b8d76263ca042acf12f: Status 404 returned error can't find the container with id 3eb56983d50581d2149436e92e873886e306412c9f1b9b8d76263ca042acf12f Sep 20 01:56:33.197723 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:33.197685 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b9558bfb-l72p4" event={"ID":"d92be662-6898-4c73-af02-c415ad35bcf1","Type":"ContainerStarted","Data":"8d29521681e42243e5e147f2a0b186df52a2347133909aa7cd7239b967a16e98"} Sep 20 01:56:33.198107 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:33.197742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b9558bfb-l72p4" event={"ID":"d92be662-6898-4c73-af02-c415ad35bcf1","Type":"ContainerStarted","Data":"3eb56983d50581d2149436e92e873886e306412c9f1b9b8d76263ca042acf12f"} Sep 20 01:56:33.215426 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:33.215379 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74b9558bfb-l72p4" podStartSLOduration=2.21536419 podStartE2EDuration="2.21536419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:56:33.213801768 +0000 UTC m=+244.732117526" watchObservedRunningTime="2026-09-20 01:56:33.21536419 +0000 UTC m=+244.733679946" Sep 20 01:56:42.276202 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:42.276161 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:42.276689 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:42.276226 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:42.284237 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:42.284213 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:43.238490 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:43.238465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:56:43.285306 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:56:43.285256 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:57:08.306334 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.306291 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-767d987f49-7pj74" podUID="c647ec3b-ad03-431d-a145-94f7f2d8193c" containerName="console" containerID="cri-o://eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3" gracePeriod=15 Sep 20 01:57:08.538685 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.538661 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:57:08.556432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556320 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xq7g\" (UniqueName: \"kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556369 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556417 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556468 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556551 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556650 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556572 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert\") pod \"c647ec3b-ad03-431d-a145-94f7f2d8193c\" (UID: \"c647ec3b-ad03-431d-a145-94f7f2d8193c\") " Sep 20 01:57:08.556844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556739 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:57:08.556844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.556810 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.557521 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.557140 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config" (OuterVolumeSpecName: "console-config") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:57:08.557521 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.557334 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:57:08.557521 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.557481 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:57:08.559476 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.559449 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g" (OuterVolumeSpecName: "kube-api-access-9xq7g") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "kube-api-access-9xq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:57:08.560051 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.560027 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:57:08.560424 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.560393 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c647ec3b-ad03-431d-a145-94f7f2d8193c" (UID: "c647ec3b-ad03-431d-a145-94f7f2d8193c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:57:08.657528 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657505 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.657528 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657525 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-oauth-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.657666 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657536 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-console-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.657666 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657543 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-trusted-ca-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.657666 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657551 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c647ec3b-ad03-431d-a145-94f7f2d8193c-oauth-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:08.657666 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:08.657560 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xq7g\" (UniqueName: \"kubernetes.io/projected/c647ec3b-ad03-431d-a145-94f7f2d8193c-kube-api-access-9xq7g\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:57:09.334380 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.334293 2578 generic.go:356] "Generic (PLEG): container finished" podID="c647ec3b-ad03-431d-a145-94f7f2d8193c" containerID="eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3" exitCode=0 Sep 20 01:57:09.334380 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.334358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767d987f49-7pj74" event={"ID":"c647ec3b-ad03-431d-a145-94f7f2d8193c","Type":"ContainerDied","Data":"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3"} Sep 20 01:57:09.334860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.334379 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767d987f49-7pj74" Sep 20 01:57:09.334860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.334396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767d987f49-7pj74" event={"ID":"c647ec3b-ad03-431d-a145-94f7f2d8193c","Type":"ContainerDied","Data":"1f3a1a56e0ab9dc39fec913cacc8ac1152641a54f51cb818b3925416f73fdc67"} Sep 20 01:57:09.334860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.334414 2578 scope.go:117] "RemoveContainer" containerID="eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3" Sep 20 01:57:09.342611 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.342593 2578 scope.go:117] "RemoveContainer" containerID="eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3" Sep 20 01:57:09.342887 ip-10-0-137-50 kubenswrapper[2578]: E0920 01:57:09.342861 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3\": container with ID starting with eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3 not found: ID does not exist" containerID="eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3" Sep 20 01:57:09.343033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.342894 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3"} err="failed to get container status \"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3\": rpc error: code = NotFound desc = could not find container \"eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3\": container with ID starting with eea13f9623158343cb899587ac879722fd6625a144b53fa5caf1973ea003a9d3 not found: ID does not exist" Sep 20 01:57:09.348815 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.348796 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:57:09.352544 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:09.352516 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-767d987f49-7pj74"] Sep 20 01:57:11.013731 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:11.013694 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c647ec3b-ad03-431d-a145-94f7f2d8193c" path="/var/lib/kubelet/pods/c647ec3b-ad03-431d-a145-94f7f2d8193c/volumes" Sep 20 01:57:28.913268 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:28.913240 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:57:28.919985 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:28.919964 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 20 01:57:28.923344 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:57:28.923315 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 01:58:16.118474 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.118443 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84"] Sep 20 01:58:16.118922 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.118770 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c647ec3b-ad03-431d-a145-94f7f2d8193c" containerName="console" Sep 20 01:58:16.118922 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.118781 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c647ec3b-ad03-431d-a145-94f7f2d8193c" containerName="console" Sep 20 01:58:16.118922 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.118839 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c647ec3b-ad03-431d-a145-94f7f2d8193c" containerName="console" Sep 20 01:58:16.121896 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.121878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.124561 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.124539 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 20 01:58:16.124663 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.124572 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gwn52\"" Sep 20 01:58:16.124663 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.124540 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 20 01:58:16.134229 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.134210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84"] Sep 20 01:58:16.166712 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.166684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.166828 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.166733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzf8\" (UniqueName: \"kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.166828 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.166809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.267312 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.267266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.267442 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.267322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzf8\" (UniqueName: \"kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.267442 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.267357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.267609 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.267589 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.267721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.267645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.275818 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.275789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzf8\" (UniqueName: \"kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.431313 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.431288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:16.695998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.695849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84"] Sep 20 01:58:16.698359 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:58:16.698336 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ee37de_61bd_4556_89d3_9d13b4fb4d91.slice/crio-1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab WatchSource:0}: Error finding container 1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab: Status 404 returned error can't find the container with id 1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab Sep 20 01:58:16.700326 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:16.700304 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 01:58:17.588706 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:17.588665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" event={"ID":"d9ee37de-61bd-4556-89d3-9d13b4fb4d91","Type":"ContainerStarted","Data":"1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab"} Sep 20 01:58:24.620080 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:24.620048 2578 generic.go:356] "Generic (PLEG): container finished" podID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerID="ac6df47835f85e2d0836a8f0ae053698c2644062205ec1dc93068aab32d5279d" exitCode=0 Sep 20 01:58:24.620547 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:24.620138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" event={"ID":"d9ee37de-61bd-4556-89d3-9d13b4fb4d91","Type":"ContainerDied","Data":"ac6df47835f85e2d0836a8f0ae053698c2644062205ec1dc93068aab32d5279d"} Sep 20 01:58:26.633787 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:26.633755 2578 generic.go:356] "Generic (PLEG): container finished" podID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerID="5a7c1f46b1451a6056e642e754ec34cf4815f928e2651f427753691c68116906" exitCode=0 Sep 20 01:58:26.634125 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:26.633827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" event={"ID":"d9ee37de-61bd-4556-89d3-9d13b4fb4d91","Type":"ContainerDied","Data":"5a7c1f46b1451a6056e642e754ec34cf4815f928e2651f427753691c68116906"} Sep 20 01:58:33.667249 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:33.667212 2578 generic.go:356] "Generic (PLEG): container finished" podID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerID="cc16d2dc454f58b5266289fb26559ecb8dfad96ecb2205121f461e2ba6c8a7a2" exitCode=0 Sep 20 01:58:33.667609 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:33.667297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" event={"ID":"d9ee37de-61bd-4556-89d3-9d13b4fb4d91","Type":"ContainerDied","Data":"cc16d2dc454f58b5266289fb26559ecb8dfad96ecb2205121f461e2ba6c8a7a2"} Sep 20 01:58:34.792164 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.792144 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:34.913510 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.913474 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle\") pod \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " Sep 20 01:58:34.913659 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.913564 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzf8\" (UniqueName: \"kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8\") pod \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " Sep 20 01:58:34.913659 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.913601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util\") pod \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\" (UID: \"d9ee37de-61bd-4556-89d3-9d13b4fb4d91\") " Sep 20 01:58:34.914266 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.914237 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle" (OuterVolumeSpecName: "bundle") pod "d9ee37de-61bd-4556-89d3-9d13b4fb4d91" (UID: "d9ee37de-61bd-4556-89d3-9d13b4fb4d91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:58:34.915895 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.915872 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8" (OuterVolumeSpecName: "kube-api-access-xlzf8") pod "d9ee37de-61bd-4556-89d3-9d13b4fb4d91" (UID: "d9ee37de-61bd-4556-89d3-9d13b4fb4d91"). InnerVolumeSpecName "kube-api-access-xlzf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:58:34.917583 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:34.917563 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util" (OuterVolumeSpecName: "util") pod "d9ee37de-61bd-4556-89d3-9d13b4fb4d91" (UID: "d9ee37de-61bd-4556-89d3-9d13b4fb4d91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:58:35.014560 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.014495 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:35.014560 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.014518 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzf8\" (UniqueName: \"kubernetes.io/projected/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-kube-api-access-xlzf8\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:35.014560 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.014526 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9ee37de-61bd-4556-89d3-9d13b4fb4d91-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:35.675944 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.675892 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" Sep 20 01:58:35.676109 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.675898 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754097v84" event={"ID":"d9ee37de-61bd-4556-89d3-9d13b4fb4d91","Type":"ContainerDied","Data":"1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab"} Sep 20 01:58:35.676109 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:35.676027 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fba7cc2901c401afa96f263dda4d972211a5b96de6cd4cdaa84edc5adb25eab" Sep 20 01:58:39.015761 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.015729 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm"] Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016042 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="pull" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016054 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="pull" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016066 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="extract" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016071 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="extract" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016079 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="util" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016084 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="util" Sep 20 01:58:39.016217 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.016129 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ee37de-61bd-4556-89d3-9d13b4fb4d91" containerName="extract" Sep 20 01:58:39.058177 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.058139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm"] Sep 20 01:58:39.058341 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.058287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.060829 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.060801 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dlcv4\"" Sep 20 01:58:39.063732 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.063320 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 20 01:58:39.063732 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.063533 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 20 01:58:39.146440 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.146408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a462839-4c02-4605-9d44-fce3ca86b150-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.146440 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.146451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpbh\" (UniqueName: \"kubernetes.io/projected/0a462839-4c02-4605-9d44-fce3ca86b150-kube-api-access-jmpbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.247860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.247826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a462839-4c02-4605-9d44-fce3ca86b150-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.247860 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.247866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpbh\" (UniqueName: \"kubernetes.io/projected/0a462839-4c02-4605-9d44-fce3ca86b150-kube-api-access-jmpbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.248255 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.248236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a462839-4c02-4605-9d44-fce3ca86b150-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.255743 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.255720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpbh\" (UniqueName: \"kubernetes.io/projected/0a462839-4c02-4605-9d44-fce3ca86b150-kube-api-access-jmpbh\") pod \"cert-manager-operator-controller-manager-55c444bf86-jvtsm\" (UID: \"0a462839-4c02-4605-9d44-fce3ca86b150\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.375204 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.375128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" Sep 20 01:58:39.664884 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.664853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm"] Sep 20 01:58:39.668046 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:58:39.668020 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a462839_4c02_4605_9d44_fce3ca86b150.slice/crio-21a31a893d96df252631706a42fb17c6ddb4836abf88a193cfbee6f9e7f6786a WatchSource:0}: Error finding container 21a31a893d96df252631706a42fb17c6ddb4836abf88a193cfbee6f9e7f6786a: Status 404 returned error can't find the container with id 21a31a893d96df252631706a42fb17c6ddb4836abf88a193cfbee6f9e7f6786a Sep 20 01:58:39.694028 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:39.694007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" event={"ID":"0a462839-4c02-4605-9d44-fce3ca86b150","Type":"ContainerStarted","Data":"21a31a893d96df252631706a42fb17c6ddb4836abf88a193cfbee6f9e7f6786a"} Sep 20 01:58:42.708775 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:42.708734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" event={"ID":"0a462839-4c02-4605-9d44-fce3ca86b150","Type":"ContainerStarted","Data":"68b3ee38e20a8698af13fd125bebf7d680cb35147c7631b12aa5819b649f8852"} Sep 20 01:58:42.760345 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:42.760291 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jvtsm" podStartSLOduration=2.164152122 podStartE2EDuration="4.760275307s" totalImagesPullingTime="2.596123185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:58:42.75546346 +0000 UTC m=+374.273779252" watchObservedRunningTime="2026-09-20 01:58:42.760275307 +0000 UTC m=+374.278591064" Sep 20 01:58:43.501705 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.501676 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g"] Sep 20 01:58:43.519016 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.518993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.523927 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.523886 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gwn52\"" Sep 20 01:58:43.524595 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.524579 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 20 01:58:43.524770 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.524754 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 20 01:58:43.529133 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.529112 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g"] Sep 20 01:58:43.583448 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.583421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.583599 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.583472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.583599 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.583553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxqh\" (UniqueName: \"kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.593124 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.593099 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jnm44"] Sep 20 01:58:43.599474 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.599456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.601746 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.601727 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-q8b2x\"" Sep 20 01:58:43.601837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.601751 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 20 01:58:43.601837 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.601731 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 20 01:58:43.607536 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.607518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jnm44"] Sep 20 01:58:43.684808 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.684780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.684967 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.684824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.684967 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.684851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxqh\" (UniqueName: \"kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.684967 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.684928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cj98\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-kube-api-access-9cj98\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.685076 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.684973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.685183 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.685167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.685287 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.685271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.692990 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.692965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxqh\" (UniqueName: \"kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.785470 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.785394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.785470 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.785440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cj98\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-kube-api-access-9cj98\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.798745 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.798712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.798840 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.798814 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cj98\" (UniqueName: \"kubernetes.io/projected/51927308-7e0f-40f1-8650-c7600e73070c-kube-api-access-9cj98\") pod \"cert-manager-webhook-6486ffd997-jnm44\" (UID: \"51927308-7e0f-40f1-8650-c7600e73070c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:43.828612 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.828590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:43.917064 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:43.917036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:44.096987 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.096967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g"] Sep 20 01:58:44.099041 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:58:44.099019 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382b9970_9f65_4d44_a3ca_ccf0988c5c3b.slice/crio-19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280 WatchSource:0}: Error finding container 19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280: Status 404 returned error can't find the container with id 19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280 Sep 20 01:58:44.183064 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.182983 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jnm44"] Sep 20 01:58:44.186068 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:58:44.186045 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51927308_7e0f_40f1_8650_c7600e73070c.slice/crio-08ece23d17f0dcd7bc676ad117925b54d30653555381b8a0b2829fb346a63b46 WatchSource:0}: Error finding container 08ece23d17f0dcd7bc676ad117925b54d30653555381b8a0b2829fb346a63b46: Status 404 returned error can't find the container with id 08ece23d17f0dcd7bc676ad117925b54d30653555381b8a0b2829fb346a63b46 Sep 20 01:58:44.720121 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.720085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" event={"ID":"51927308-7e0f-40f1-8650-c7600e73070c","Type":"ContainerStarted","Data":"08ece23d17f0dcd7bc676ad117925b54d30653555381b8a0b2829fb346a63b46"} Sep 20 01:58:44.721850 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.721826 2578 generic.go:356] "Generic (PLEG): container finished" podID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerID="7abf95ef57efd0e3ec99a6fe648f9b1c68c3f85114a0384ebfd36d1aa126118d" exitCode=0 Sep 20 01:58:44.722011 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.721944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" event={"ID":"382b9970-9f65-4d44-a3ca-ccf0988c5c3b","Type":"ContainerDied","Data":"7abf95ef57efd0e3ec99a6fe648f9b1c68c3f85114a0384ebfd36d1aa126118d"} Sep 20 01:58:44.722011 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:44.721975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" event={"ID":"382b9970-9f65-4d44-a3ca-ccf0988c5c3b","Type":"ContainerStarted","Data":"19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280"} Sep 20 01:58:46.733018 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:46.732974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" event={"ID":"51927308-7e0f-40f1-8650-c7600e73070c","Type":"ContainerStarted","Data":"d157f5613868c8c582965354dc0017a4b72e2b8ce36b7e9bdda19a0897931683"} Sep 20 01:58:46.733380 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:46.733120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:58:46.748419 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:46.748374 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" podStartSLOduration=1.368143173 podStartE2EDuration="3.748360436s" totalImagesPullingTime="2.380217263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:58:46.748158736 +0000 UTC m=+378.266474484" watchObservedRunningTime="2026-09-20 01:58:46.748360436 +0000 UTC m=+378.266676193" Sep 20 01:58:48.201492 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.201416 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-njc85"] Sep 20 01:58:48.203382 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.203364 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.205509 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.205493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kw4cp\"" Sep 20 01:58:48.211649 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.211628 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-njc85"] Sep 20 01:58:48.323127 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.323090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.323290 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.323147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdcf\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-kube-api-access-8kdcf\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.424284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.424253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.424448 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.424304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdcf\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-kube-api-access-8kdcf\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.433918 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.433870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.434056 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.434040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdcf\" (UniqueName: \"kubernetes.io/projected/10868d6e-b515-4e51-8d13-f97791c0b3fb-kube-api-access-8kdcf\") pod \"cert-manager-cainjector-559cdd8c87-njc85\" (UID: \"10868d6e-b515-4e51-8d13-f97791c0b3fb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.512959 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.512883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" Sep 20 01:58:48.745516 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.745487 2578 generic.go:356] "Generic (PLEG): container finished" podID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerID="fb3572ebdf861814c0a06d62a89a74960668197f2aca79ce6807ef9aa3f378a8" exitCode=0 Sep 20 01:58:48.745693 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.745571 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" event={"ID":"382b9970-9f65-4d44-a3ca-ccf0988c5c3b","Type":"ContainerDied","Data":"fb3572ebdf861814c0a06d62a89a74960668197f2aca79ce6807ef9aa3f378a8"} Sep 20 01:58:48.783864 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:48.783808 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-njc85"] Sep 20 01:58:48.785847 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:58:48.785822 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10868d6e_b515_4e51_8d13_f97791c0b3fb.slice/crio-751cf286a39fdda2b478159b2f5ad0d0bfb4858f17413e4d77e7edba5fe33a2b WatchSource:0}: Error finding container 751cf286a39fdda2b478159b2f5ad0d0bfb4858f17413e4d77e7edba5fe33a2b: Status 404 returned error can't find the container with id 751cf286a39fdda2b478159b2f5ad0d0bfb4858f17413e4d77e7edba5fe33a2b Sep 20 01:58:49.753251 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:49.753213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" event={"ID":"10868d6e-b515-4e51-8d13-f97791c0b3fb","Type":"ContainerStarted","Data":"ad5d30da177dcd7e27d57e6e97b928bb4e90ab3eee3cd1d190ed60db3b52685e"} Sep 20 01:58:49.753668 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:49.753268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" event={"ID":"10868d6e-b515-4e51-8d13-f97791c0b3fb","Type":"ContainerStarted","Data":"751cf286a39fdda2b478159b2f5ad0d0bfb4858f17413e4d77e7edba5fe33a2b"} Sep 20 01:58:49.755415 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:49.755390 2578 generic.go:356] "Generic (PLEG): container finished" podID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerID="304cb2ffb44252dccc3fe6f6442a3833ef43854f4abab78e9aa152bb87d1c92f" exitCode=0 Sep 20 01:58:49.755528 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:49.755443 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" event={"ID":"382b9970-9f65-4d44-a3ca-ccf0988c5c3b","Type":"ContainerDied","Data":"304cb2ffb44252dccc3fe6f6442a3833ef43854f4abab78e9aa152bb87d1c92f"} Sep 20 01:58:49.771786 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:49.771745 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-njc85" podStartSLOduration=1.771733768 podStartE2EDuration="1.771733768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:58:49.770863218 +0000 UTC m=+381.289178986" watchObservedRunningTime="2026-09-20 01:58:49.771733768 +0000 UTC m=+381.290049524" Sep 20 01:58:50.885156 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.885125 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:50.945821 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.945788 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util\") pod \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " Sep 20 01:58:50.946004 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.945840 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle\") pod \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " Sep 20 01:58:50.946004 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.945880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsxqh\" (UniqueName: \"kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh\") pod \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\" (UID: \"382b9970-9f65-4d44-a3ca-ccf0988c5c3b\") " Sep 20 01:58:50.946316 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.946287 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle" (OuterVolumeSpecName: "bundle") pod "382b9970-9f65-4d44-a3ca-ccf0988c5c3b" (UID: "382b9970-9f65-4d44-a3ca-ccf0988c5c3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:58:50.948179 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.948153 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh" (OuterVolumeSpecName: "kube-api-access-dsxqh") pod "382b9970-9f65-4d44-a3ca-ccf0988c5c3b" (UID: "382b9970-9f65-4d44-a3ca-ccf0988c5c3b"). InnerVolumeSpecName "kube-api-access-dsxqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:58:50.950466 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:50.950442 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util" (OuterVolumeSpecName: "util") pod "382b9970-9f65-4d44-a3ca-ccf0988c5c3b" (UID: "382b9970-9f65-4d44-a3ca-ccf0988c5c3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:58:51.047013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.046986 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:51.047013 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.047010 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxqh\" (UniqueName: \"kubernetes.io/projected/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-kube-api-access-dsxqh\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:51.047174 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.047020 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382b9970-9f65-4d44-a3ca-ccf0988c5c3b-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:58:51.766440 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.766406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" event={"ID":"382b9970-9f65-4d44-a3ca-ccf0988c5c3b","Type":"ContainerDied","Data":"19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280"} Sep 20 01:58:51.766440 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.766431 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkpt7g" Sep 20 01:58:51.766638 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:51.766457 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19782efc65e3aa0abd9a6e503f16ed178c6fc38144254548466b431bacad1280" Sep 20 01:58:52.741224 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:58:52.741197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jnm44" Sep 20 01:59:01.154171 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154138 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c"] Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154459 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="pull" Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154470 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="pull" Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154485 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="extract" Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154495 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="extract" Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154508 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="util" Sep 20 01:59:01.154534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154513 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="util" Sep 20 01:59:01.154719 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.154572 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="382b9970-9f65-4d44-a3ca-ccf0988c5c3b" containerName="extract" Sep 20 01:59:01.157522 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.157505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.159742 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.159721 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 20 01:59:01.160532 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.160517 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 20 01:59:01.160607 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.160538 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hgs6f\"" Sep 20 01:59:01.164465 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.164445 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c"] Sep 20 01:59:01.330680 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.330650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.330842 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.330690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wd9\" (UniqueName: \"kubernetes.io/projected/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-kube-api-access-k6wd9\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.431307 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.431213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.431307 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.431271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wd9\" (UniqueName: \"kubernetes.io/projected/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-kube-api-access-k6wd9\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.431578 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.431561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.438721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.438703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wd9\" (UniqueName: \"kubernetes.io/projected/b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e-kube-api-access-k6wd9\") pod \"openshift-lws-operator-fd8ccff4c-w7g5c\" (UID: \"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.467333 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.467309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" Sep 20 01:59:01.736219 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.736196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c"] Sep 20 01:59:01.738593 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:01.738566 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb27a791e_cf05_4f3a_a7f9_7f2eb8320d8e.slice/crio-32c20cd42d941dd96785a0667c3f27396133323f8e38930d1f7847f843aad314 WatchSource:0}: Error finding container 32c20cd42d941dd96785a0667c3f27396133323f8e38930d1f7847f843aad314: Status 404 returned error can't find the container with id 32c20cd42d941dd96785a0667c3f27396133323f8e38930d1f7847f843aad314 Sep 20 01:59:01.812241 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:01.812206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" event={"ID":"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e","Type":"ContainerStarted","Data":"32c20cd42d941dd96785a0667c3f27396133323f8e38930d1f7847f843aad314"} Sep 20 01:59:04.828404 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:04.828328 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" event={"ID":"b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e","Type":"ContainerStarted","Data":"76d8e380f2f90465bb3aad4cf69952f2e10c365666ae2ca0d761dfbb8771aaed"} Sep 20 01:59:04.844602 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:04.844548 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w7g5c" podStartSLOduration=1.061981669 podStartE2EDuration="3.844532571s" totalImagesPullingTime="2.782550902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:59:04.842519942 +0000 UTC m=+396.360835698" watchObservedRunningTime="2026-09-20 01:59:04.844532571 +0000 UTC m=+396.362848327" Sep 20 01:59:07.282930 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.282875 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-788d4985c7-24xc5"] Sep 20 01:59:07.286264 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.286246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.296802 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.296775 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788d4985c7-24xc5"] Sep 20 01:59:07.381400 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381368 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-service-ca\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wclg\" (UniqueName: \"kubernetes.io/projected/dca76d6a-b579-44ca-94b0-17b4830ee7e8-kube-api-access-5wclg\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-oauth-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-oauth-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-trusted-ca-bundle\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.381594 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.381575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482631 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-trusted-ca-bundle\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482631 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482838 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-service-ca\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482838 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wclg\" (UniqueName: \"kubernetes.io/projected/dca76d6a-b579-44ca-94b0-17b4830ee7e8-kube-api-access-5wclg\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482838 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-oauth-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482838 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-oauth-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.482838 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.482766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.483412 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.483384 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.483686 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.483486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-service-ca\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.483686 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.483566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-trusted-ca-bundle\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.483686 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.483566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dca76d6a-b579-44ca-94b0-17b4830ee7e8-oauth-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.485348 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.485322 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-oauth-config\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.485457 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.485415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dca76d6a-b579-44ca-94b0-17b4830ee7e8-console-serving-cert\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.490415 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.490394 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wclg\" (UniqueName: \"kubernetes.io/projected/dca76d6a-b579-44ca-94b0-17b4830ee7e8-kube-api-access-5wclg\") pod \"console-788d4985c7-24xc5\" (UID: \"dca76d6a-b579-44ca-94b0-17b4830ee7e8\") " pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.594574 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.594508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:07.865005 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:07.864983 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788d4985c7-24xc5"] Sep 20 01:59:07.867337 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:07.867310 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca76d6a_b579_44ca_94b0_17b4830ee7e8.slice/crio-acc2634651bba9fb842baafa7a712a54e8e8aec93f39c764f64f7636ea9af662 WatchSource:0}: Error finding container acc2634651bba9fb842baafa7a712a54e8e8aec93f39c764f64f7636ea9af662: Status 404 returned error can't find the container with id acc2634651bba9fb842baafa7a712a54e8e8aec93f39c764f64f7636ea9af662 Sep 20 01:59:08.847816 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.847779 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788d4985c7-24xc5" event={"ID":"dca76d6a-b579-44ca-94b0-17b4830ee7e8","Type":"ContainerStarted","Data":"1c9af79c4e8edda4c96936de3857a2d1565f4a6a8fdc397b053ff0ccf3329f72"} Sep 20 01:59:08.848292 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.847838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788d4985c7-24xc5" event={"ID":"dca76d6a-b579-44ca-94b0-17b4830ee7e8","Type":"ContainerStarted","Data":"acc2634651bba9fb842baafa7a712a54e8e8aec93f39c764f64f7636ea9af662"} Sep 20 01:59:08.863862 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.863815 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-788d4985c7-24xc5" podStartSLOduration=1.8638005130000002 podStartE2EDuration="1.863800513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:59:08.863060218 +0000 UTC m=+400.381375975" watchObservedRunningTime="2026-09-20 01:59:08.863800513 +0000 UTC m=+400.382116269" Sep 20 01:59:08.926320 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.926291 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst"] Sep 20 01:59:08.929719 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.929704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:08.932074 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.932054 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 20 01:59:08.932165 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.932102 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gwn52\"" Sep 20 01:59:08.932165 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.932055 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 20 01:59:08.937094 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:08.937076 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst"] Sep 20 01:59:09.096872 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.096842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.097068 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.096881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.097068 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.096987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgztw\" (UniqueName: \"kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.197603 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.197578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgztw\" (UniqueName: \"kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.197752 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.197620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.197752 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.197642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.197957 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.197939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.198005 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.197993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.205721 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.205697 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgztw\" (UniqueName: \"kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.240227 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.240196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:09.507994 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.507970 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst"] Sep 20 01:59:09.509617 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:09.509589 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c16944_ba07_4de2_b386_3b24bf9049bb.slice/crio-88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384 WatchSource:0}: Error finding container 88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384: Status 404 returned error can't find the container with id 88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384 Sep 20 01:59:09.853429 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.853355 2578 generic.go:356] "Generic (PLEG): container finished" podID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerID="6c8f76c47d067b456728d874e63d00e776ef72c5c84a3b9de93b3023553ffbfa" exitCode=0 Sep 20 01:59:09.853977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.853437 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" event={"ID":"94c16944-ba07-4de2-b386-3b24bf9049bb","Type":"ContainerDied","Data":"6c8f76c47d067b456728d874e63d00e776ef72c5c84a3b9de93b3023553ffbfa"} Sep 20 01:59:09.853977 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:09.853484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" event={"ID":"94c16944-ba07-4de2-b386-3b24bf9049bb","Type":"ContainerStarted","Data":"88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384"} Sep 20 01:59:10.860386 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:10.860354 2578 generic.go:356] "Generic (PLEG): container finished" podID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerID="d775291b4aca06a737fc30326ef8f4db1a63c4dc82c8e2a3a0fb84fcd9a09fa6" exitCode=0 Sep 20 01:59:10.860809 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:10.860439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" event={"ID":"94c16944-ba07-4de2-b386-3b24bf9049bb","Type":"ContainerDied","Data":"d775291b4aca06a737fc30326ef8f4db1a63c4dc82c8e2a3a0fb84fcd9a09fa6"} Sep 20 01:59:11.867031 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:11.866997 2578 generic.go:356] "Generic (PLEG): container finished" podID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerID="00f9e3b517051064448e80d7d974093a6fc4c8b9b58d731afd811274da4dffaf" exitCode=0 Sep 20 01:59:11.867414 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:11.867087 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" event={"ID":"94c16944-ba07-4de2-b386-3b24bf9049bb","Type":"ContainerDied","Data":"00f9e3b517051064448e80d7d974093a6fc4c8b9b58d731afd811274da4dffaf"} Sep 20 01:59:12.999431 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:12.999407 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:13.031196 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.031169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util\") pod \"94c16944-ba07-4de2-b386-3b24bf9049bb\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " Sep 20 01:59:13.031308 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.031210 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle\") pod \"94c16944-ba07-4de2-b386-3b24bf9049bb\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " Sep 20 01:59:13.032248 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.032224 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle" (OuterVolumeSpecName: "bundle") pod "94c16944-ba07-4de2-b386-3b24bf9049bb" (UID: "94c16944-ba07-4de2-b386-3b24bf9049bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:59:13.036880 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.036861 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util" (OuterVolumeSpecName: "util") pod "94c16944-ba07-4de2-b386-3b24bf9049bb" (UID: "94c16944-ba07-4de2-b386-3b24bf9049bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:59:13.132592 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.132520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgztw\" (UniqueName: \"kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw\") pod \"94c16944-ba07-4de2-b386-3b24bf9049bb\" (UID: \"94c16944-ba07-4de2-b386-3b24bf9049bb\") " Sep 20 01:59:13.132717 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.132659 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:13.132717 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.132671 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94c16944-ba07-4de2-b386-3b24bf9049bb-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:13.134600 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.134572 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw" (OuterVolumeSpecName: "kube-api-access-kgztw") pod "94c16944-ba07-4de2-b386-3b24bf9049bb" (UID: "94c16944-ba07-4de2-b386-3b24bf9049bb"). InnerVolumeSpecName "kube-api-access-kgztw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:59:13.233423 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.233383 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgztw\" (UniqueName: \"kubernetes.io/projected/94c16944-ba07-4de2-b386-3b24bf9049bb-kube-api-access-kgztw\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:13.878007 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.877984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" Sep 20 01:59:13.878167 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.877981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsvst" event={"ID":"94c16944-ba07-4de2-b386-3b24bf9049bb","Type":"ContainerDied","Data":"88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384"} Sep 20 01:59:13.878167 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:13.878098 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88cfac0ee224035131a3bc17dcbab4a6f52621c38bba0313ac4f776d0754d384" Sep 20 01:59:17.595309 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:17.595272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:17.595724 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:17.595330 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:17.599534 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:17.599516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:17.899495 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:17.899470 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-788d4985c7-24xc5" Sep 20 01:59:17.961404 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:17.961376 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:59:18.925657 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925622 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg"] Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925960 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="extract" Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925971 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="extract" Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925979 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="pull" Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925984 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="pull" Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925995 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="util" Sep 20 01:59:18.926044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.925999 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="util" Sep 20 01:59:18.926245 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.926053 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c16944-ba07-4de2-b386-3b24bf9049bb" containerName="extract" Sep 20 01:59:18.931325 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.931309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:18.933494 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.933469 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 20 01:59:18.934329 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.934309 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gwn52\"" Sep 20 01:59:18.934432 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.934315 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 20 01:59:18.936899 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.936881 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg"] Sep 20 01:59:18.975800 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.975769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:18.975947 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.975840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:18.975947 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:18.975897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d6p\" (UniqueName: \"kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.076606 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.076578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.076755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.076615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d6p\" (UniqueName: \"kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.076755 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.076676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.077021 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.076998 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.077070 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.077010 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.090378 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.090352 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d6p\" (UniqueName: \"kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.241511 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.241430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:19.513044 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.513020 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg"] Sep 20 01:59:19.515411 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:19.515385 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cacd3d_caeb_4af2_bf1d_8c45fe13239f.slice/crio-a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b WatchSource:0}: Error finding container a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b: Status 404 returned error can't find the container with id a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b Sep 20 01:59:19.906444 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.906415 2578 generic.go:356] "Generic (PLEG): container finished" podID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerID="1fe02dadc2d82313eeb96b8ef764c1fd26df9e17138e8e32b6ce936402cca0a8" exitCode=0 Sep 20 01:59:19.906643 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.906506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" event={"ID":"08cacd3d-caeb-4af2-bf1d-8c45fe13239f","Type":"ContainerDied","Data":"1fe02dadc2d82313eeb96b8ef764c1fd26df9e17138e8e32b6ce936402cca0a8"} Sep 20 01:59:19.906643 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:19.906553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" event={"ID":"08cacd3d-caeb-4af2-bf1d-8c45fe13239f","Type":"ContainerStarted","Data":"a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b"} Sep 20 01:59:20.160454 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.160385 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5"] Sep 20 01:59:20.163850 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.163830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.167212 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.167196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 20 01:59:20.167426 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.167405 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 20 01:59:20.167551 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.167534 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 20 01:59:20.167682 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.167658 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9h799\"" Sep 20 01:59:20.183876 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.183853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5"] Sep 20 01:59:20.185100 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.185081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.185177 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.185111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/acc11168-00d0-4003-b954-9681c8a9cf77-manager-config\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.185177 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.185129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-metrics-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.185258 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.185204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tc7v\" (UniqueName: \"kubernetes.io/projected/acc11168-00d0-4003-b954-9681c8a9cf77-kube-api-access-5tc7v\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.286493 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.286458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.286493 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.286500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/acc11168-00d0-4003-b954-9681c8a9cf77-manager-config\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.286737 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.286515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-metrics-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.286737 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.286558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tc7v\" (UniqueName: \"kubernetes.io/projected/acc11168-00d0-4003-b954-9681c8a9cf77-kube-api-access-5tc7v\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.287201 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.287178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/acc11168-00d0-4003-b954-9681c8a9cf77-manager-config\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.289181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.289153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-metrics-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.289181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.289160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acc11168-00d0-4003-b954-9681c8a9cf77-cert\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.294697 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.294672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tc7v\" (UniqueName: \"kubernetes.io/projected/acc11168-00d0-4003-b954-9681c8a9cf77-kube-api-access-5tc7v\") pod \"lws-controller-manager-6bdfb9dff9-t6vx5\" (UID: \"acc11168-00d0-4003-b954-9681c8a9cf77\") " pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.472674 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.472648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:20.747865 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.747839 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5"] Sep 20 01:59:20.750199 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:20.750172 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc11168_00d0_4003_b954_9681c8a9cf77.slice/crio-5749a803a3194f89f4e15fc61ea91d40e0f1012f0f6a2558cfb076ee4d19ff96 WatchSource:0}: Error finding container 5749a803a3194f89f4e15fc61ea91d40e0f1012f0f6a2558cfb076ee4d19ff96: Status 404 returned error can't find the container with id 5749a803a3194f89f4e15fc61ea91d40e0f1012f0f6a2558cfb076ee4d19ff96 Sep 20 01:59:20.912801 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.912762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" event={"ID":"acc11168-00d0-4003-b954-9681c8a9cf77","Type":"ContainerStarted","Data":"5749a803a3194f89f4e15fc61ea91d40e0f1012f0f6a2558cfb076ee4d19ff96"} Sep 20 01:59:20.914662 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.914640 2578 generic.go:356] "Generic (PLEG): container finished" podID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerID="9c9efda13da56f210e23e3b956d428e20ef679f5bb44438459fe78cc8ac2dda5" exitCode=0 Sep 20 01:59:20.914787 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:20.914699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" event={"ID":"08cacd3d-caeb-4af2-bf1d-8c45fe13239f","Type":"ContainerDied","Data":"9c9efda13da56f210e23e3b956d428e20ef679f5bb44438459fe78cc8ac2dda5"} Sep 20 01:59:21.044268 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.044195 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7"] Sep 20 01:59:21.048708 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.048691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.052316 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.052289 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 20 01:59:21.052647 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.052627 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 20 01:59:21.053253 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.053231 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 20 01:59:21.053562 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.053545 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l7wm4\"" Sep 20 01:59:21.053856 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.053833 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 20 01:59:21.074514 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.074490 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7"] Sep 20 01:59:21.093171 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.093133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-tmp\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.093307 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.093215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-manifests\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.093366 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.093319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.093417 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.093358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czxt\" (UniqueName: \"kubernetes.io/projected/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-kube-api-access-7czxt\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.093417 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.093395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.193998 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.193961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-manifests\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.194428 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.194027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.194428 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.194050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czxt\" (UniqueName: \"kubernetes.io/projected/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-kube-api-access-7czxt\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.194428 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.194073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.194428 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.194104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-tmp\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.194428 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.194355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-manifests\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.196498 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.196473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-tmp\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.196740 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.196721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.196833 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.196815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.205260 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.205236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czxt\" (UniqueName: \"kubernetes.io/projected/534ccdc6-9d44-4e44-97a6-ba0b734bff6c-kube-api-access-7czxt\") pod \"opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7\" (UID: \"534ccdc6-9d44-4e44-97a6-ba0b734bff6c\") " pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.363652 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.363562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:21.678870 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.678843 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7"] Sep 20 01:59:21.679998 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:21.679970 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534ccdc6_9d44_4e44_97a6_ba0b734bff6c.slice/crio-f210a427f34cdeb07e0e90f242101c055ae0158c648bd194dc0d15ded0b41dd2 WatchSource:0}: Error finding container f210a427f34cdeb07e0e90f242101c055ae0158c648bd194dc0d15ded0b41dd2: Status 404 returned error can't find the container with id f210a427f34cdeb07e0e90f242101c055ae0158c648bd194dc0d15ded0b41dd2 Sep 20 01:59:21.922564 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.922528 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" event={"ID":"534ccdc6-9d44-4e44-97a6-ba0b734bff6c","Type":"ContainerStarted","Data":"f210a427f34cdeb07e0e90f242101c055ae0158c648bd194dc0d15ded0b41dd2"} Sep 20 01:59:21.924955 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.924926 2578 generic.go:356] "Generic (PLEG): container finished" podID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerID="9af90d12f7d346372b44b693c1ea279c1bbcba81fe78831e32ab98cee71560fd" exitCode=0 Sep 20 01:59:21.925078 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:21.925016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" event={"ID":"08cacd3d-caeb-4af2-bf1d-8c45fe13239f","Type":"ContainerDied","Data":"9af90d12f7d346372b44b693c1ea279c1bbcba81fe78831e32ab98cee71560fd"} Sep 20 01:59:23.941991 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:23.941953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" event={"ID":"acc11168-00d0-4003-b954-9681c8a9cf77","Type":"ContainerStarted","Data":"31bb39749ea3d30b22e61b67fbb9b160e6adb747f5b89cae72d3cbcddb165dcf"} Sep 20 01:59:23.942435 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:23.942123 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:23.958381 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:23.958328 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" podStartSLOduration=1.85422513 podStartE2EDuration="3.958312158s" totalImagesPullingTime="2.104087028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:59:23.956607706 +0000 UTC m=+415.474923469" watchObservedRunningTime="2026-09-20 01:59:23.958312158 +0000 UTC m=+415.476627914" Sep 20 01:59:24.591100 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.591076 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:24.628033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.628001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle\") pod \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " Sep 20 01:59:24.628181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.628118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util\") pod \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " Sep 20 01:59:24.628181 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.628146 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9d6p\" (UniqueName: \"kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p\") pod \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\" (UID: \"08cacd3d-caeb-4af2-bf1d-8c45fe13239f\") " Sep 20 01:59:24.628809 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.628779 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle" (OuterVolumeSpecName: "bundle") pod "08cacd3d-caeb-4af2-bf1d-8c45fe13239f" (UID: "08cacd3d-caeb-4af2-bf1d-8c45fe13239f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:59:24.630311 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.630285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p" (OuterVolumeSpecName: "kube-api-access-f9d6p") pod "08cacd3d-caeb-4af2-bf1d-8c45fe13239f" (UID: "08cacd3d-caeb-4af2-bf1d-8c45fe13239f"). InnerVolumeSpecName "kube-api-access-f9d6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:59:24.723302 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.723273 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util" (OuterVolumeSpecName: "util") pod "08cacd3d-caeb-4af2-bf1d-8c45fe13239f" (UID: "08cacd3d-caeb-4af2-bf1d-8c45fe13239f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 01:59:24.728885 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.728868 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:24.728885 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.728886 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9d6p\" (UniqueName: \"kubernetes.io/projected/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-kube-api-access-f9d6p\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:24.729038 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.728895 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08cacd3d-caeb-4af2-bf1d-8c45fe13239f-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:24.953293 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.953256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" event={"ID":"534ccdc6-9d44-4e44-97a6-ba0b734bff6c","Type":"ContainerStarted","Data":"a042d7e18bde343212fe0000ddb9acfeb63d5a3d0ca6de4e2b5aca871115f8e1"} Sep 20 01:59:24.955145 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.955128 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" Sep 20 01:59:24.955240 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.955167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zq9zg" event={"ID":"08cacd3d-caeb-4af2-bf1d-8c45fe13239f","Type":"ContainerDied","Data":"a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b"} Sep 20 01:59:24.955240 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:24.955202 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a67f807fe3fb57ffad5832a854c541c8b3df5264f02895977e26b51d4b2b544b" Sep 20 01:59:25.961646 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:25.961613 2578 generic.go:356] "Generic (PLEG): container finished" podID="534ccdc6-9d44-4e44-97a6-ba0b734bff6c" containerID="a042d7e18bde343212fe0000ddb9acfeb63d5a3d0ca6de4e2b5aca871115f8e1" exitCode=0 Sep 20 01:59:25.962095 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:25.961666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" event={"ID":"534ccdc6-9d44-4e44-97a6-ba0b734bff6c","Type":"ContainerDied","Data":"a042d7e18bde343212fe0000ddb9acfeb63d5a3d0ca6de4e2b5aca871115f8e1"} Sep 20 01:59:26.970831 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:26.970792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" event={"ID":"534ccdc6-9d44-4e44-97a6-ba0b734bff6c","Type":"ContainerStarted","Data":"414db08694ef31b2c57111f8efabbc85b8fbf0c29a74d453ccfe6782cd2dc35e"} Sep 20 01:59:26.971284 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:26.970942 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:26.992550 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:26.992484 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" podStartSLOduration=2.581594016 podStartE2EDuration="6.992467316s" totalImagesPullingTime="3.365320784s" totalInitContainerRuntime="1.045552516s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:59:26.99089805 +0000 UTC m=+418.509213808" watchObservedRunningTime="2026-09-20 01:59:26.992467316 +0000 UTC m=+418.510783099" Sep 20 01:59:34.957184 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:34.957151 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bdfb9dff9-t6vx5" Sep 20 01:59:37.977267 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:37.977240 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7" Sep 20 01:59:41.140562 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140531 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb"] Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140929 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="util" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140945 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="util" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140959 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="pull" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140968 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="pull" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140979 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="extract" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.140988 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="extract" Sep 20 01:59:41.141140 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.141078 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cacd3d-caeb-4af2-bf1d-8c45fe13239f" containerName="extract" Sep 20 01:59:41.150037 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.150019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.155844 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.155820 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 20 01:59:41.155987 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.155820 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 20 01:59:41.157125 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.156820 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 20 01:59:41.157125 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.156967 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6s4sl\"" Sep 20 01:59:41.157279 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.157255 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 20 01:59:41.162434 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.162411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb"] Sep 20 01:59:41.269033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.268999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tk2\" (UniqueName: \"kubernetes.io/projected/1d7fb564-48f7-467e-8406-54554da0ac65-kube-api-access-96tk2\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.269033 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.269034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d7fb564-48f7-467e-8406-54554da0ac65-tmp\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.269220 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.269124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7fb564-48f7-467e-8406-54554da0ac65-tls-certs\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.370254 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.370218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tk2\" (UniqueName: \"kubernetes.io/projected/1d7fb564-48f7-467e-8406-54554da0ac65-kube-api-access-96tk2\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.370254 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.370256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d7fb564-48f7-467e-8406-54554da0ac65-tmp\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.370491 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.370354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7fb564-48f7-467e-8406-54554da0ac65-tls-certs\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.372711 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.372681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d7fb564-48f7-467e-8406-54554da0ac65-tmp\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.372991 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.372972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7fb564-48f7-467e-8406-54554da0ac65-tls-certs\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.379330 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.379312 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tk2\" (UniqueName: \"kubernetes.io/projected/1d7fb564-48f7-467e-8406-54554da0ac65-kube-api-access-96tk2\") pod \"kube-auth-proxy-86bbc588c6-8rmmb\" (UID: \"1d7fb564-48f7-467e-8406-54554da0ac65\") " pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.460945 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.460852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" Sep 20 01:59:41.731444 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:41.731422 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb"] Sep 20 01:59:41.734004 ip-10-0-137-50 kubenswrapper[2578]: W0920 01:59:41.733968 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7fb564_48f7_467e_8406_54554da0ac65.slice/crio-d6993175606c5c6f53cec51b3de8ad8b875369c13f137c01726d06e8a8bc83c3 WatchSource:0}: Error finding container d6993175606c5c6f53cec51b3de8ad8b875369c13f137c01726d06e8a8bc83c3: Status 404 returned error can't find the container with id d6993175606c5c6f53cec51b3de8ad8b875369c13f137c01726d06e8a8bc83c3 Sep 20 01:59:42.040996 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:42.040899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" event={"ID":"1d7fb564-48f7-467e-8406-54554da0ac65","Type":"ContainerStarted","Data":"d6993175606c5c6f53cec51b3de8ad8b875369c13f137c01726d06e8a8bc83c3"} Sep 20 01:59:42.981995 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:42.981934 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74b9558bfb-l72p4" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" containerName="console" containerID="cri-o://8d29521681e42243e5e147f2a0b186df52a2347133909aa7cd7239b967a16e98" gracePeriod=15 Sep 20 01:59:44.062470 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.062430 2578 generic.go:356] "Generic (PLEG): container finished" podID="d92be662-6898-4c73-af02-c415ad35bcf1" containerID="8d29521681e42243e5e147f2a0b186df52a2347133909aa7cd7239b967a16e98" exitCode=0 Sep 20 01:59:44.062942 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.062512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b9558bfb-l72p4" event={"ID":"d92be662-6898-4c73-af02-c415ad35bcf1","Type":"ContainerDied","Data":"8d29521681e42243e5e147f2a0b186df52a2347133909aa7cd7239b967a16e98"} Sep 20 01:59:44.236122 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.236074 2578 patch_prober.go:28] interesting pod/console-74b9558bfb-l72p4 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.20:8443/health\": context deadline exceeded" start-of-body= Sep 20 01:59:44.236290 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.236152 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-74b9558bfb-l72p4" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": context deadline exceeded" Sep 20 01:59:44.483043 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.483022 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:59:44.602115 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602088 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602257 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602128 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602257 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602257 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602422 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602288 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602422 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602422 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkf4t\" (UniqueName: \"kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t\") pod \"d92be662-6898-4c73-af02-c415ad35bcf1\" (UID: \"d92be662-6898-4c73-af02-c415ad35bcf1\") " Sep 20 01:59:44.602592 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602568 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:59:44.602648 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602604 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca" (OuterVolumeSpecName: "service-ca") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:59:44.602648 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602634 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config" (OuterVolumeSpecName: "console-config") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:59:44.602749 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.602719 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 01:59:44.604330 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.604308 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:59:44.604446 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.604429 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 01:59:44.604780 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.604759 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t" (OuterVolumeSpecName: "kube-api-access-tkf4t") pod "d92be662-6898-4c73-af02-c415ad35bcf1" (UID: "d92be662-6898-4c73-af02-c415ad35bcf1"). InnerVolumeSpecName "kube-api-access-tkf4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 01:59:44.703839 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703802 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.703839 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703832 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-console-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.703839 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703844 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.704087 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703857 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-oauth-serving-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.704087 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703869 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkf4t\" (UniqueName: \"kubernetes.io/projected/d92be662-6898-4c73-af02-c415ad35bcf1-kube-api-access-tkf4t\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.704087 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703880 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d92be662-6898-4c73-af02-c415ad35bcf1-console-oauth-config\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:44.704087 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:44.703890 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d92be662-6898-4c73-af02-c415ad35bcf1-trusted-ca-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 01:59:45.073699 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.073660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" event={"ID":"1d7fb564-48f7-467e-8406-54554da0ac65","Type":"ContainerStarted","Data":"4ab98299cd480f847db79b596f3d46310cc95f6d609bf46b303ee95454a4e6fb"} Sep 20 01:59:45.075484 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.075457 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b9558bfb-l72p4" Sep 20 01:59:45.075614 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.075515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b9558bfb-l72p4" event={"ID":"d92be662-6898-4c73-af02-c415ad35bcf1","Type":"ContainerDied","Data":"3eb56983d50581d2149436e92e873886e306412c9f1b9b8d76263ca042acf12f"} Sep 20 01:59:45.075614 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.075557 2578 scope.go:117] "RemoveContainer" containerID="8d29521681e42243e5e147f2a0b186df52a2347133909aa7cd7239b967a16e98" Sep 20 01:59:45.094382 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.094327 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86bbc588c6-8rmmb" podStartSLOduration=1.313390672 podStartE2EDuration="4.094306138s" totalImagesPullingTime="2.780915466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 01:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 01:59:45.08932683 +0000 UTC m=+436.607642587" watchObservedRunningTime="2026-09-20 01:59:45.094306138 +0000 UTC m=+436.612621896" Sep 20 01:59:45.106620 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.106577 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:59:45.112092 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:45.112065 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74b9558bfb-l72p4"] Sep 20 01:59:47.014668 ip-10-0-137-50 kubenswrapper[2578]: I0920 01:59:47.014633 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" path="/var/lib/kubelet/pods/d92be662-6898-4c73-af02-c415ad35bcf1/volumes" Sep 20 02:01:03.115717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.115667 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4"] Sep 20 02:01:03.116384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.116172 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" containerName="console" Sep 20 02:01:03.116384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.116191 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" containerName="console" Sep 20 02:01:03.116384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.116304 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92be662-6898-4c73-af02-c415ad35bcf1" containerName="console" Sep 20 02:01:03.119816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.119797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.122059 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.122035 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 20 02:01:03.122201 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.122100 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 20 02:01:03.122730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.122716 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gwn52\"" Sep 20 02:01:03.127776 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.127757 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4"] Sep 20 02:01:03.216089 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.216057 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24"] Sep 20 02:01:03.219522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.219507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.227333 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.227309 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24"] Sep 20 02:01:03.238584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.238557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4x4\" (UniqueName: \"kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.238686 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.238608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.238686 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.238659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.314765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.314729 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq"] Sep 20 02:01:03.318255 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.318240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.326145 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.326120 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq"] Sep 20 02:01:03.340129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.340293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.340293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.340293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.340423 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4x4\" (UniqueName: \"kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.340423 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzhl\" (UniqueName: \"kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.340521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.340521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.340469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.348127 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.348102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4x4\" (UniqueName: \"kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.415527 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.415488 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz"] Sep 20 02:01:03.419195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.419175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.431302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.431272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:03.431606 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.431586 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz"] Sep 20 02:01:03.441210 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.441301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.441301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzhl\" (UniqueName: \"kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.441441 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.441441 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgx44\" (UniqueName: \"kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.441540 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.441603 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441591 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.441866 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.441844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.449115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.449066 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzhl\" (UniqueName: \"kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.529125 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.529094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:03.542164 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.542270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.542270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.542270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.542437 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgx44\" (UniqueName: \"kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.542437 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cmj\" (UniqueName: \"kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.542549 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.542619 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.542557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.550604 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.550581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgx44\" (UniqueName: \"kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.628381 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.628351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:03.643547 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.643515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.643653 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.643569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cmj\" (UniqueName: \"kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.643653 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.643604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.643876 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.643855 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.643954 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.643900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.653385 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.653360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cmj\" (UniqueName: \"kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.704204 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.704178 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4"] Sep 20 02:01:03.707275 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:01:03.707251 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39c66cb_3460_45bd_8ef7_1eeebf261530.slice/crio-3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57 WatchSource:0}: Error finding container 3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57: Status 404 returned error can't find the container with id 3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57 Sep 20 02:01:03.733834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.733814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:03.806432 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.806404 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24"] Sep 20 02:01:03.808413 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:01:03.808372 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bf432c_fe9a_4261_88a7_c052183d782e.slice/crio-3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f WatchSource:0}: Error finding container 3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f: Status 404 returned error can't find the container with id 3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f Sep 20 02:01:03.910642 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:03.910615 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq"] Sep 20 02:01:03.911430 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:01:03.911398 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e254e3_adeb_4765_a02b_31fcb2c8626d.slice/crio-fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef WatchSource:0}: Error finding container fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef: Status 404 returned error can't find the container with id fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef Sep 20 02:01:04.012831 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.012730 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz"] Sep 20 02:01:04.070662 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:01:04.070634 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2b759b_cb44_4ea3_a264_21ef863085ef.slice/crio-6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4 WatchSource:0}: Error finding container 6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4: Status 404 returned error can't find the container with id 6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4 Sep 20 02:01:04.459471 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.459437 2578 generic.go:356] "Generic (PLEG): container finished" podID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerID="f52400e16afb4fcb5629d38bb77e14f0411de121aec874f64fb7d8b93db06cf6" exitCode=0 Sep 20 02:01:04.459872 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.459521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerDied","Data":"f52400e16afb4fcb5629d38bb77e14f0411de121aec874f64fb7d8b93db06cf6"} Sep 20 02:01:04.459872 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.459570 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerStarted","Data":"3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57"} Sep 20 02:01:04.461135 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.461095 2578 generic.go:356] "Generic (PLEG): container finished" podID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerID="fc0425d19089707f7dadb108a558eee785a749e73d805529c8d59a4356bcad3b" exitCode=0 Sep 20 02:01:04.461183 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.461175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerDied","Data":"fc0425d19089707f7dadb108a558eee785a749e73d805529c8d59a4356bcad3b"} Sep 20 02:01:04.461224 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.461214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerStarted","Data":"6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4"} Sep 20 02:01:04.462747 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.462728 2578 generic.go:356] "Generic (PLEG): container finished" podID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerID="7e14105d4467cc9eab6175172f5153e196c00ce1c6d8d950b994a348d9b732d3" exitCode=0 Sep 20 02:01:04.462839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.462785 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerDied","Data":"7e14105d4467cc9eab6175172f5153e196c00ce1c6d8d950b994a348d9b732d3"} Sep 20 02:01:04.462839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.462813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerStarted","Data":"fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef"} Sep 20 02:01:04.464494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.464473 2578 generic.go:356] "Generic (PLEG): container finished" podID="38bf432c-fe9a-4261-88a7-c052183d782e" containerID="804cfbf7188ac9541e80617bd5120e5a6b03bbd333d46da9f20d7e61a21e1972" exitCode=0 Sep 20 02:01:04.464631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.464519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerDied","Data":"804cfbf7188ac9541e80617bd5120e5a6b03bbd333d46da9f20d7e61a21e1972"} Sep 20 02:01:04.464631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:04.464552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerStarted","Data":"3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f"} Sep 20 02:01:05.474548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:05.474451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerStarted","Data":"a9c6edc676f292f931c50c32785e7475946ad4b0c6958ffc97750407fa95ef39"} Sep 20 02:01:05.477174 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:05.477143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerStarted","Data":"7b6d6f992b2ce17c631dad59cc49adfe7eae8dcd4c7a8aa208aa369391288447"} Sep 20 02:01:05.479435 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:05.479393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerStarted","Data":"6f609c416104e0960558f48387f229b0e61e546a8871f865d511729677dd1cbd"} Sep 20 02:01:05.482213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:05.482191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerStarted","Data":"e864905db302302a748ea1261be8ff46a4ccc1002b7ebda03aa66e7df47419a2"} Sep 20 02:01:06.489202 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.489166 2578 generic.go:356] "Generic (PLEG): container finished" podID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerID="7b6d6f992b2ce17c631dad59cc49adfe7eae8dcd4c7a8aa208aa369391288447" exitCode=0 Sep 20 02:01:06.489630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.489255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerDied","Data":"7b6d6f992b2ce17c631dad59cc49adfe7eae8dcd4c7a8aa208aa369391288447"} Sep 20 02:01:06.491461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.491440 2578 generic.go:356] "Generic (PLEG): container finished" podID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerID="6f609c416104e0960558f48387f229b0e61e546a8871f865d511729677dd1cbd" exitCode=0 Sep 20 02:01:06.491461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.491459 2578 generic.go:356] "Generic (PLEG): container finished" podID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerID="f9dcfe6ff5526b6527dae7a08327188abb54e5b15b754dd97dae4a7cf5a60b36" exitCode=0 Sep 20 02:01:06.491793 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.491510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerDied","Data":"6f609c416104e0960558f48387f229b0e61e546a8871f865d511729677dd1cbd"} Sep 20 02:01:06.491793 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.491544 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerDied","Data":"f9dcfe6ff5526b6527dae7a08327188abb54e5b15b754dd97dae4a7cf5a60b36"} Sep 20 02:01:06.493627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.493605 2578 generic.go:356] "Generic (PLEG): container finished" podID="38bf432c-fe9a-4261-88a7-c052183d782e" containerID="e864905db302302a748ea1261be8ff46a4ccc1002b7ebda03aa66e7df47419a2" exitCode=0 Sep 20 02:01:06.493627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.493626 2578 generic.go:356] "Generic (PLEG): container finished" podID="38bf432c-fe9a-4261-88a7-c052183d782e" containerID="416b94be137a22fed3cf53cf2004580a3e995cb050ca748d611eea4e5789b453" exitCode=0 Sep 20 02:01:06.493787 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.493681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerDied","Data":"e864905db302302a748ea1261be8ff46a4ccc1002b7ebda03aa66e7df47419a2"} Sep 20 02:01:06.493787 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.493719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerDied","Data":"416b94be137a22fed3cf53cf2004580a3e995cb050ca748d611eea4e5789b453"} Sep 20 02:01:06.495512 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.495493 2578 generic.go:356] "Generic (PLEG): container finished" podID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerID="a9c6edc676f292f931c50c32785e7475946ad4b0c6958ffc97750407fa95ef39" exitCode=0 Sep 20 02:01:06.495601 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:06.495566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerDied","Data":"a9c6edc676f292f931c50c32785e7475946ad4b0c6958ffc97750407fa95ef39"} Sep 20 02:01:07.502748 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.502713 2578 generic.go:356] "Generic (PLEG): container finished" podID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerID="078973244b0c447c662761a3cd42cd31e8f72d2c62b49ef6ab937c71c4422d5c" exitCode=0 Sep 20 02:01:07.503153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.502794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerDied","Data":"078973244b0c447c662761a3cd42cd31e8f72d2c62b49ef6ab937c71c4422d5c"} Sep 20 02:01:07.504922 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.504882 2578 generic.go:356] "Generic (PLEG): container finished" podID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerID="fc5ed2cbac925a08e23d597b31ec9c4082ee72d4fd2202b4847b953f49376acb" exitCode=0 Sep 20 02:01:07.505095 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.504938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerDied","Data":"fc5ed2cbac925a08e23d597b31ec9c4082ee72d4fd2202b4847b953f49376acb"} Sep 20 02:01:07.633891 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.633865 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:07.660713 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.660692 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:07.781729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util\") pod \"38bf432c-fe9a-4261-88a7-c052183d782e\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " Sep 20 02:01:07.781729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle\") pod \"38bf432c-fe9a-4261-88a7-c052183d782e\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " Sep 20 02:01:07.781729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781718 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util\") pod \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " Sep 20 02:01:07.782052 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle\") pod \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " Sep 20 02:01:07.782052 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781769 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djzhl\" (UniqueName: \"kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl\") pod \"38bf432c-fe9a-4261-88a7-c052183d782e\" (UID: \"38bf432c-fe9a-4261-88a7-c052183d782e\") " Sep 20 02:01:07.782052 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.781789 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgx44\" (UniqueName: \"kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44\") pod \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\" (UID: \"f6e254e3-adeb-4765-a02b-31fcb2c8626d\") " Sep 20 02:01:07.782256 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.782230 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle" (OuterVolumeSpecName: "bundle") pod "f6e254e3-adeb-4765-a02b-31fcb2c8626d" (UID: "f6e254e3-adeb-4765-a02b-31fcb2c8626d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:07.782360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.782339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle" (OuterVolumeSpecName: "bundle") pod "38bf432c-fe9a-4261-88a7-c052183d782e" (UID: "38bf432c-fe9a-4261-88a7-c052183d782e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:07.784058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.784034 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl" (OuterVolumeSpecName: "kube-api-access-djzhl") pod "38bf432c-fe9a-4261-88a7-c052183d782e" (UID: "38bf432c-fe9a-4261-88a7-c052183d782e"). InnerVolumeSpecName "kube-api-access-djzhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:01:07.784392 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.784373 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44" (OuterVolumeSpecName: "kube-api-access-jgx44") pod "f6e254e3-adeb-4765-a02b-31fcb2c8626d" (UID: "f6e254e3-adeb-4765-a02b-31fcb2c8626d"). InnerVolumeSpecName "kube-api-access-jgx44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:01:07.787840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.787817 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util" (OuterVolumeSpecName: "util") pod "f6e254e3-adeb-4765-a02b-31fcb2c8626d" (UID: "f6e254e3-adeb-4765-a02b-31fcb2c8626d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:07.789053 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.789032 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util" (OuterVolumeSpecName: "util") pod "38bf432c-fe9a-4261-88a7-c052183d782e" (UID: "38bf432c-fe9a-4261-88a7-c052183d782e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:07.882664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882629 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzhl\" (UniqueName: \"kubernetes.io/projected/38bf432c-fe9a-4261-88a7-c052183d782e-kube-api-access-djzhl\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:07.882664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882658 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgx44\" (UniqueName: \"kubernetes.io/projected/f6e254e3-adeb-4765-a02b-31fcb2c8626d-kube-api-access-jgx44\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:07.882664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882667 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:07.882664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882675 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38bf432c-fe9a-4261-88a7-c052183d782e-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:07.882957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882682 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:07.882957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:07.882690 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e254e3-adeb-4765-a02b-31fcb2c8626d-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.511250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.511212 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" event={"ID":"f6e254e3-adeb-4765-a02b-31fcb2c8626d","Type":"ContainerDied","Data":"fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef"} Sep 20 02:01:08.511250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.511233 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpfwq" Sep 20 02:01:08.511250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.511258 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde922d4d5929f364120b57e2061de9f8ddf90c9a069b02cce6d2b5386d626ef" Sep 20 02:01:08.513093 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.513076 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" Sep 20 02:01:08.513203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.513118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccgv24" event={"ID":"38bf432c-fe9a-4261-88a7-c052183d782e","Type":"ContainerDied","Data":"3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f"} Sep 20 02:01:08.513203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.513154 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7526647ecd841291415b6144ab329c1ca4d41c4e1ee32e058f5225a75e430f" Sep 20 02:01:08.639936 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.639897 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:08.679103 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.679081 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:08.791452 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle\") pod \"b39c66cb-3460-45bd-8ef7-1eeebf261530\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " Sep 20 02:01:08.791452 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791431 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7cmj\" (UniqueName: \"kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj\") pod \"7c2b759b-cb44-4ea3-a264-21ef863085ef\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " Sep 20 02:01:08.791671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util\") pod \"b39c66cb-3460-45bd-8ef7-1eeebf261530\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " Sep 20 02:01:08.791671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle\") pod \"7c2b759b-cb44-4ea3-a264-21ef863085ef\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " Sep 20 02:01:08.791671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791541 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss4x4\" (UniqueName: \"kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4\") pod \"b39c66cb-3460-45bd-8ef7-1eeebf261530\" (UID: \"b39c66cb-3460-45bd-8ef7-1eeebf261530\") " Sep 20 02:01:08.791671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.791580 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util\") pod \"7c2b759b-cb44-4ea3-a264-21ef863085ef\" (UID: \"7c2b759b-cb44-4ea3-a264-21ef863085ef\") " Sep 20 02:01:08.792045 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.792020 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle" (OuterVolumeSpecName: "bundle") pod "b39c66cb-3460-45bd-8ef7-1eeebf261530" (UID: "b39c66cb-3460-45bd-8ef7-1eeebf261530"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:08.792548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.792519 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle" (OuterVolumeSpecName: "bundle") pod "7c2b759b-cb44-4ea3-a264-21ef863085ef" (UID: "7c2b759b-cb44-4ea3-a264-21ef863085ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:08.793801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.793780 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4" (OuterVolumeSpecName: "kube-api-access-ss4x4") pod "b39c66cb-3460-45bd-8ef7-1eeebf261530" (UID: "b39c66cb-3460-45bd-8ef7-1eeebf261530"). InnerVolumeSpecName "kube-api-access-ss4x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:01:08.794031 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.794013 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj" (OuterVolumeSpecName: "kube-api-access-m7cmj") pod "7c2b759b-cb44-4ea3-a264-21ef863085ef" (UID: "7c2b759b-cb44-4ea3-a264-21ef863085ef"). InnerVolumeSpecName "kube-api-access-m7cmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:01:08.797179 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.797155 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util" (OuterVolumeSpecName: "util") pod "b39c66cb-3460-45bd-8ef7-1eeebf261530" (UID: "b39c66cb-3460-45bd-8ef7-1eeebf261530"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:08.797746 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.797727 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util" (OuterVolumeSpecName: "util") pod "7c2b759b-cb44-4ea3-a264-21ef863085ef" (UID: "7c2b759b-cb44-4ea3-a264-21ef863085ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:01:08.892843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892814 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7cmj\" (UniqueName: \"kubernetes.io/projected/7c2b759b-cb44-4ea3-a264-21ef863085ef-kube-api-access-m7cmj\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.892843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892839 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.892843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892850 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.893058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892858 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4x4\" (UniqueName: \"kubernetes.io/projected/b39c66cb-3460-45bd-8ef7-1eeebf261530-kube-api-access-ss4x4\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.893058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892867 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2b759b-cb44-4ea3-a264-21ef863085ef-util\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:08.893058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:08.892875 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c66cb-3460-45bd-8ef7-1eeebf261530-bundle\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:01:09.520207 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.520180 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" Sep 20 02:01:09.520634 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.520207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fxs4" event={"ID":"b39c66cb-3460-45bd-8ef7-1eeebf261530","Type":"ContainerDied","Data":"3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57"} Sep 20 02:01:09.520634 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.520255 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b63299a0f659d194e7630932244e4cb08451d3b361296dbdac0b21f887fdd57" Sep 20 02:01:09.522196 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.522170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" event={"ID":"7c2b759b-cb44-4ea3-a264-21ef863085ef","Type":"ContainerDied","Data":"6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4"} Sep 20 02:01:09.522350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.522203 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awfhpz" Sep 20 02:01:09.522350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:09.522214 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd464985fda9fec2a0130b412f26c8dab2d50150e54ac92f9318627c91cebd4" Sep 20 02:01:27.472299 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472227 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk"] Sep 20 02:01:27.472703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472687 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="pull" Sep 20 02:01:27.472703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472706 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="pull" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472718 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472726 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472734 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472741 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472748 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="extract" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472753 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="extract" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472760 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="pull" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472765 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="pull" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472770 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="extract" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472775 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="extract" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472783 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472788 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472797 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472802 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="util" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472809 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="pull" Sep 20 02:01:27.472803 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472814 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="pull" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472820 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472824 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472835 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472840 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472851 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="pull" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="pull" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472935 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38bf432c-fe9a-4261-88a7-c052183d782e" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472946 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2b759b-cb44-4ea3-a264-21ef863085ef" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472952 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e254e3-adeb-4765-a02b-31fcb2c8626d" containerName="extract" Sep 20 02:01:27.473336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.472960 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39c66cb-3460-45bd-8ef7-1eeebf261530" containerName="extract" Sep 20 02:01:27.475689 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.475674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.477879 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.477856 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 20 02:01:27.478017 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.477895 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 20 02:01:27.478017 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.477940 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dxnrh\"" Sep 20 02:01:27.478635 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.478621 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 20 02:01:27.478675 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.478623 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 20 02:01:27.483418 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.483400 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk"] Sep 20 02:01:27.541052 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.541013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.541193 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.541113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/459ff040-e649-4123-abd3-4d1afdfbc599-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.541193 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.541137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkh4\" (UniqueName: \"kubernetes.io/projected/459ff040-e649-4123-abd3-4d1afdfbc599-kube-api-access-bdkh4\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.641996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.641965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/459ff040-e649-4123-abd3-4d1afdfbc599-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.641996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.642000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkh4\" (UniqueName: \"kubernetes.io/projected/459ff040-e649-4123-abd3-4d1afdfbc599-kube-api-access-bdkh4\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.642213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.642041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.642213 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:01:27.642147 2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 20 02:01:27.642213 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:01:27.642213 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert podName:459ff040-e649-4123-abd3-4d1afdfbc599 nodeName:}" failed. No retries permitted until 2026-09-20 02:01:28.142193189 +0000 UTC m=+539.660508941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-ttzmk" (UID: "459ff040-e649-4123-abd3-4d1afdfbc599") : secret "plugin-serving-cert" not found Sep 20 02:01:27.642648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.642628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/459ff040-e649-4123-abd3-4d1afdfbc599-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:27.652043 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:27.652017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkh4\" (UniqueName: \"kubernetes.io/projected/459ff040-e649-4123-abd3-4d1afdfbc599-kube-api-access-bdkh4\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:28.147411 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:28.147374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:28.149876 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:28.149851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/459ff040-e649-4123-abd3-4d1afdfbc599-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ttzmk\" (UID: \"459ff040-e649-4123-abd3-4d1afdfbc599\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:28.385645 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:28.385613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" Sep 20 02:01:28.656066 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:28.656042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk"] Sep 20 02:01:28.657771 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:01:28.657736 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459ff040_e649_4123_abd3_4d1afdfbc599.slice/crio-7acbb3c261c68b070d410dd7be381bf50d29af4f136f80d8864759bcf7bb9625 WatchSource:0}: Error finding container 7acbb3c261c68b070d410dd7be381bf50d29af4f136f80d8864759bcf7bb9625: Status 404 returned error can't find the container with id 7acbb3c261c68b070d410dd7be381bf50d29af4f136f80d8864759bcf7bb9625 Sep 20 02:01:29.636051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:29.636013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" event={"ID":"459ff040-e649-4123-abd3-4d1afdfbc599","Type":"ContainerStarted","Data":"7acbb3c261c68b070d410dd7be381bf50d29af4f136f80d8864759bcf7bb9625"} Sep 20 02:01:33.661122 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:33.661086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" event={"ID":"459ff040-e649-4123-abd3-4d1afdfbc599","Type":"ContainerStarted","Data":"aeef9af4f835654c0c99b8eab4b4d2dbb2bde6540c446dd3093d5667893bd259"} Sep 20 02:01:33.677342 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:01:33.677295 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ttzmk" podStartSLOduration=1.774122299 podStartE2EDuration="6.677280169s" totalImagesPullingTime="4.90315787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:01:33.675742561 +0000 UTC m=+545.194058331" watchObservedRunningTime="2026-09-20 02:01:33.677280169 +0000 UTC m=+545.195595925" Sep 20 02:02:24.993083 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:24.993051 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:02:24.995492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:24.995476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:24.997545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:24.997518 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 20 02:02:25.006748 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.006726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:02:25.018081 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.018059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:02:25.040587 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.040566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd375002-74ad-475a-ab0d-f694454c9026-config-file\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.040686 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.040619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndk4\" (UniqueName: \"kubernetes.io/projected/fd375002-74ad-475a-ab0d-f694454c9026-kube-api-access-jndk4\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.142038 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.142011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd375002-74ad-475a-ab0d-f694454c9026-config-file\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.142175 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.142058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndk4\" (UniqueName: \"kubernetes.io/projected/fd375002-74ad-475a-ab0d-f694454c9026-kube-api-access-jndk4\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.142619 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.142595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd375002-74ad-475a-ab0d-f694454c9026-config-file\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.149521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.149499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndk4\" (UniqueName: \"kubernetes.io/projected/fd375002-74ad-475a-ab0d-f694454c9026-kube-api-access-jndk4\") pod \"limitador-limitador-84994f7ffc-mcfb4\" (UID: \"fd375002-74ad-475a-ab0d-f694454c9026\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.306588 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.306514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:25.412340 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.412308 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:25.414740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.414720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:25.417172 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.417151 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gnzt4\"" Sep 20 02:02:25.422704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.422680 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:25.445026 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.444996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4lk\" (UniqueName: \"kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk\") pod \"authorino-7bcfd8f9cd-gvgnz\" (UID: \"b9724aac-dc3f-42cf-a51f-2f6572dbfe55\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:25.545621 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.545589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4lk\" (UniqueName: \"kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk\") pod \"authorino-7bcfd8f9cd-gvgnz\" (UID: \"b9724aac-dc3f-42cf-a51f-2f6572dbfe55\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:25.553523 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.553497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4lk\" (UniqueName: \"kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk\") pod \"authorino-7bcfd8f9cd-gvgnz\" (UID: \"b9724aac-dc3f-42cf-a51f-2f6572dbfe55\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:25.575861 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.575841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:02:25.577853 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:02:25.577819 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd375002_74ad_475a_ab0d_f694454c9026.slice/crio-1102231911c908f15587b7f62d5b7fa41fd23c22a60dc32e4dcdce64d08fdaed WatchSource:0}: Error finding container 1102231911c908f15587b7f62d5b7fa41fd23c22a60dc32e4dcdce64d08fdaed: Status 404 returned error can't find the container with id 1102231911c908f15587b7f62d5b7fa41fd23c22a60dc32e4dcdce64d08fdaed Sep 20 02:02:25.725295 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.725265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:25.929219 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.929189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" event={"ID":"fd375002-74ad-475a-ab0d-f694454c9026","Type":"ContainerStarted","Data":"1102231911c908f15587b7f62d5b7fa41fd23c22a60dc32e4dcdce64d08fdaed"} Sep 20 02:02:25.993982 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:25.993958 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:25.995418 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:02:25.995393 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9724aac_dc3f_42cf_a51f_2f6572dbfe55.slice/crio-e4b560f349a73dbb5b997f3a7760226d6656a96139b8b0c8550e31131345c045 WatchSource:0}: Error finding container e4b560f349a73dbb5b997f3a7760226d6656a96139b8b0c8550e31131345c045: Status 404 returned error can't find the container with id e4b560f349a73dbb5b997f3a7760226d6656a96139b8b0c8550e31131345c045 Sep 20 02:02:26.938203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:26.938164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" event={"ID":"b9724aac-dc3f-42cf-a51f-2f6572dbfe55","Type":"ContainerStarted","Data":"e4b560f349a73dbb5b997f3a7760226d6656a96139b8b0c8550e31131345c045"} Sep 20 02:02:28.950478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.950439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" event={"ID":"b9724aac-dc3f-42cf-a51f-2f6572dbfe55","Type":"ContainerStarted","Data":"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3"} Sep 20 02:02:28.951962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.951930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" event={"ID":"fd375002-74ad-475a-ab0d-f694454c9026","Type":"ContainerStarted","Data":"57493a771a5f1aed2ad58e614debb7d616c012cda0b8bcd4bf1c460cc2e22987"} Sep 20 02:02:28.952114 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.952021 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:28.958576 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.958558 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:02:28.960650 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.960618 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:02:28.967054 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.967017 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" podStartSLOduration=1.179818785 podStartE2EDuration="3.967004027s" totalImagesPullingTime="2.787185242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:02:28.964659257 +0000 UTC m=+600.482975019" watchObservedRunningTime="2026-09-20 02:02:28.967004027 +0000 UTC m=+600.485319784" Sep 20 02:02:28.982229 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:28.982179 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" podStartSLOduration=1.7813389929999999 podStartE2EDuration="4.982165496s" totalImagesPullingTime="3.200826503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:02:28.97953583 +0000 UTC m=+600.497851588" watchObservedRunningTime="2026-09-20 02:02:28.982165496 +0000 UTC m=+600.500481254" Sep 20 02:02:30.041257 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:30.041226 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:30.962682 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:30.962626 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" podUID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" containerName="authorino" containerID="cri-o://b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3" gracePeriod=30 Sep 20 02:02:31.214972 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.214884 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:31.306276 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.306240 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr4lk\" (UniqueName: \"kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk\") pod \"b9724aac-dc3f-42cf-a51f-2f6572dbfe55\" (UID: \"b9724aac-dc3f-42cf-a51f-2f6572dbfe55\") " Sep 20 02:02:31.308455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.308431 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk" (OuterVolumeSpecName: "kube-api-access-qr4lk") pod "b9724aac-dc3f-42cf-a51f-2f6572dbfe55" (UID: "b9724aac-dc3f-42cf-a51f-2f6572dbfe55"). InnerVolumeSpecName "kube-api-access-qr4lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:02:31.407720 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.407691 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4lk\" (UniqueName: \"kubernetes.io/projected/b9724aac-dc3f-42cf-a51f-2f6572dbfe55-kube-api-access-qr4lk\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:02:31.968546 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.968514 2578 generic.go:356] "Generic (PLEG): container finished" podID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" containerID="b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3" exitCode=0 Sep 20 02:02:31.968724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.968558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" Sep 20 02:02:31.968724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.968580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" event={"ID":"b9724aac-dc3f-42cf-a51f-2f6572dbfe55","Type":"ContainerDied","Data":"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3"} Sep 20 02:02:31.968724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.968617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gvgnz" event={"ID":"b9724aac-dc3f-42cf-a51f-2f6572dbfe55","Type":"ContainerDied","Data":"e4b560f349a73dbb5b997f3a7760226d6656a96139b8b0c8550e31131345c045"} Sep 20 02:02:31.968724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.968640 2578 scope.go:117] "RemoveContainer" containerID="b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3" Sep 20 02:02:31.977692 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.977676 2578 scope.go:117] "RemoveContainer" containerID="b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3" Sep 20 02:02:31.977965 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:02:31.977947 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3\": container with ID starting with b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3 not found: ID does not exist" containerID="b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3" Sep 20 02:02:31.978035 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.977979 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3"} err="failed to get container status \"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3\": rpc error: code = NotFound desc = could not find container \"b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3\": container with ID starting with b07a08585d4c3aa818e94bc6e2d17cd766cc24173ed4c4c5bda9049e04e2aab3 not found: ID does not exist" Sep 20 02:02:31.989313 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.989290 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:31.992472 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:31.992451 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gvgnz"] Sep 20 02:02:33.015271 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:33.015240 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" path="/var/lib/kubelet/pods/b9724aac-dc3f-42cf-a51f-2f6572dbfe55/volumes" Sep 20 02:02:39.958835 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:39.958801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mcfb4" Sep 20 02:02:57.714982 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.714887 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:02:57.715379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.715241 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" containerName="authorino" Sep 20 02:02:57.715379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.715252 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" containerName="authorino" Sep 20 02:02:57.715379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.715367 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9724aac-dc3f-42cf-a51f-2f6572dbfe55" containerName="authorino" Sep 20 02:02:57.718465 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.718451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:57.720525 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.720508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gnzt4\"" Sep 20 02:02:57.723760 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.723740 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:02:57.729019 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.728993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65sc6\" (UniqueName: \"kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6\") pod \"authorino-557f876f69-qnlwp\" (UID: \"6a80790b-155c-4881-a0e2-6ea8fc5069f2\") " pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:57.830099 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.830072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65sc6\" (UniqueName: \"kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6\") pod \"authorino-557f876f69-qnlwp\" (UID: \"6a80790b-155c-4881-a0e2-6ea8fc5069f2\") " pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:57.840068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.840050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65sc6\" (UniqueName: \"kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6\") pod \"authorino-557f876f69-qnlwp\" (UID: \"6a80790b-155c-4881-a0e2-6ea8fc5069f2\") " pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:57.845454 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.845430 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:02:57.845717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.845698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:57.869086 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.869058 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p7p8w"] Sep 20 02:02:57.872958 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.872937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:57.875072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.875047 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 20 02:02:57.880602 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.880573 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p7p8w"] Sep 20 02:02:57.930722 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.930693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4vf\" (UniqueName: \"kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:57.930876 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.930857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:57.957862 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.957834 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p7p8w"] Sep 20 02:02:57.958102 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:02:57.958081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mh4vf tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" podUID="0b2baaf1-8d38-4ae7-877b-b26fc1104e9f" Sep 20 02:02:57.984838 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.984778 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:02:57.988106 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.988092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:57.994218 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:57.994193 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:02:58.031348 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.031318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.031470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.031357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vf\" (UniqueName: \"kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.031470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.031378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2n7l\" (UniqueName: \"kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.031470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.031413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.033783 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.033761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.038292 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.038269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4vf\" (UniqueName: \"kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf\") pod \"authorino-6d4dfc649f-p7p8w\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.098272 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.098247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.105463 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.105444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:58.119734 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.119715 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:02:58.121499 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:02:58.121477 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a80790b_155c_4881_a0e2_6ea8fc5069f2.slice/crio-ab474f1fece7c047abb04260eece8615fea3ca2eccd5aa309e0f8a6ed857f303 WatchSource:0}: Error finding container ab474f1fece7c047abb04260eece8615fea3ca2eccd5aa309e0f8a6ed857f303: Status 404 returned error can't find the container with id ab474f1fece7c047abb04260eece8615fea3ca2eccd5aa309e0f8a6ed857f303 Sep 20 02:02:58.131765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.131747 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh4vf\" (UniqueName: \"kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf\") pod \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " Sep 20 02:02:58.131842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.131775 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert\") pod \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\" (UID: \"0b2baaf1-8d38-4ae7-877b-b26fc1104e9f\") " Sep 20 02:02:58.132071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.132050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n7l\" (UniqueName: \"kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.132175 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.132097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.134046 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.134016 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0b2baaf1-8d38-4ae7-877b-b26fc1104e9f" (UID: "0b2baaf1-8d38-4ae7-877b-b26fc1104e9f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:02:58.134046 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.134026 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf" (OuterVolumeSpecName: "kube-api-access-mh4vf") pod "0b2baaf1-8d38-4ae7-877b-b26fc1104e9f" (UID: "0b2baaf1-8d38-4ae7-877b-b26fc1104e9f"). InnerVolumeSpecName "kube-api-access-mh4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:02:58.134347 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.134329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.139001 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.138985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n7l\" (UniqueName: \"kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l\") pod \"authorino-5ddfb79c57-bd8r4\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.232929 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.232869 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4vf\" (UniqueName: \"kubernetes.io/projected/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-kube-api-access-mh4vf\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:02:58.232929 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.232896 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f-tls-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:02:58.297307 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.297240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:02:58.567187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:58.567163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:02:58.569177 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:02:58.569155 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3ca9761_6987_47ee_b812_24fde35ed17e.slice/crio-b6cb1b2bac1f2b33264db80ffd7b23c90b88506e2e3ce794939eb9e8fc13ceff WatchSource:0}: Error finding container b6cb1b2bac1f2b33264db80ffd7b23c90b88506e2e3ce794939eb9e8fc13ceff: Status 404 returned error can't find the container with id b6cb1b2bac1f2b33264db80ffd7b23c90b88506e2e3ce794939eb9e8fc13ceff Sep 20 02:02:59.104784 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.104744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-qnlwp" event={"ID":"6a80790b-155c-4881-a0e2-6ea8fc5069f2","Type":"ContainerStarted","Data":"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b"} Sep 20 02:02:59.105166 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.104773 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-qnlwp" podUID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" containerName="authorino" containerID="cri-o://624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b" gracePeriod=30 Sep 20 02:02:59.105166 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.104809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-qnlwp" event={"ID":"6a80790b-155c-4881-a0e2-6ea8fc5069f2","Type":"ContainerStarted","Data":"ab474f1fece7c047abb04260eece8615fea3ca2eccd5aa309e0f8a6ed857f303"} Sep 20 02:02:59.106346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.106324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" event={"ID":"f3ca9761-6987-47ee-b812-24fde35ed17e","Type":"ContainerStarted","Data":"b6cb1b2bac1f2b33264db80ffd7b23c90b88506e2e3ce794939eb9e8fc13ceff"} Sep 20 02:02:59.106430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.106357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p7p8w" Sep 20 02:02:59.120015 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.119972 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-qnlwp" podStartSLOduration=1.312687443 podStartE2EDuration="2.119958724s" totalImagesPullingTime="807.271281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:02:59.118251054 +0000 UTC m=+630.636566823" watchObservedRunningTime="2026-09-20 02:02:59.119958724 +0000 UTC m=+630.638274480" Sep 20 02:02:59.141001 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.140971 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p7p8w"] Sep 20 02:02:59.144140 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.144117 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p7p8w"] Sep 20 02:02:59.337686 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.337664 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:02:59.339977 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.339961 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65sc6\" (UniqueName: \"kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6\") pod \"6a80790b-155c-4881-a0e2-6ea8fc5069f2\" (UID: \"6a80790b-155c-4881-a0e2-6ea8fc5069f2\") " Sep 20 02:02:59.341963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.341941 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6" (OuterVolumeSpecName: "kube-api-access-65sc6") pod "6a80790b-155c-4881-a0e2-6ea8fc5069f2" (UID: "6a80790b-155c-4881-a0e2-6ea8fc5069f2"). InnerVolumeSpecName "kube-api-access-65sc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:02:59.440470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:02:59.440447 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65sc6\" (UniqueName: \"kubernetes.io/projected/6a80790b-155c-4881-a0e2-6ea8fc5069f2-kube-api-access-65sc6\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:03:00.113518 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.113420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" event={"ID":"f3ca9761-6987-47ee-b812-24fde35ed17e","Type":"ContainerStarted","Data":"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922"} Sep 20 02:03:00.114657 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.114627 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" containerID="624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b" exitCode=0 Sep 20 02:03:00.114657 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.114658 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qnlwp" Sep 20 02:03:00.114816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.114705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-qnlwp" event={"ID":"6a80790b-155c-4881-a0e2-6ea8fc5069f2","Type":"ContainerDied","Data":"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b"} Sep 20 02:03:00.114816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.114747 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-qnlwp" event={"ID":"6a80790b-155c-4881-a0e2-6ea8fc5069f2","Type":"ContainerDied","Data":"ab474f1fece7c047abb04260eece8615fea3ca2eccd5aa309e0f8a6ed857f303"} Sep 20 02:03:00.114816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.114763 2578 scope.go:117] "RemoveContainer" containerID="624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b" Sep 20 02:03:00.123843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.123828 2578 scope.go:117] "RemoveContainer" containerID="624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b" Sep 20 02:03:00.124123 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:03:00.124106 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b\": container with ID starting with 624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b not found: ID does not exist" containerID="624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b" Sep 20 02:03:00.124188 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.124129 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b"} err="failed to get container status \"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b\": rpc error: code = NotFound desc = could not find container \"624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b\": container with ID starting with 624c9967cc9fefc789ec4c634854da2a3a7d808320f0d6e29aa4ed6bb1e5f49b not found: ID does not exist" Sep 20 02:03:00.146232 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.146176 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" podStartSLOduration=2.6780064059999997 podStartE2EDuration="3.146156855s" totalImagesPullingTime="468.150449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:03:00.127290412 +0000 UTC m=+631.645606164" watchObservedRunningTime="2026-09-20 02:03:00.146156855 +0000 UTC m=+631.664472615" Sep 20 02:03:00.147306 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.147281 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:03:00.149478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:00.149454 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-qnlwp"] Sep 20 02:03:01.015260 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:01.015228 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2baaf1-8d38-4ae7-877b-b26fc1104e9f" path="/var/lib/kubelet/pods/0b2baaf1-8d38-4ae7-877b-b26fc1104e9f/volumes" Sep 20 02:03:01.015445 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:01.015434 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" path="/var/lib/kubelet/pods/6a80790b-155c-4881-a0e2-6ea8fc5069f2/volumes" Sep 20 02:03:42.962192 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:03:42.962138 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:04:15.002774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.002740 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv"] Sep 20 02:04:15.003346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.003314 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" containerName="authorino" Sep 20 02:04:15.003346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.003333 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" containerName="authorino" Sep 20 02:04:15.003456 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.003421 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a80790b-155c-4881-a0e2-6ea8fc5069f2" containerName="authorino" Sep 20 02:04:15.006872 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.006850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.009208 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.009186 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ln8ck\"" Sep 20 02:04:15.009316 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.009186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 20 02:04:15.010199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.010179 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 20 02:04:15.010199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.010199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 20 02:04:15.025283 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.025260 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv"] Sep 20 02:04:15.060195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.060324 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.060324 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n779d\" (UniqueName: \"kubernetes.io/projected/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-kube-api-access-n779d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.060324 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.060493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.060493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.060433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161014 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.160986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n779d\" (UniqueName: \"kubernetes.io/projected/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-kube-api-access-n779d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161427 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161404 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161490 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.161557 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.161526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.163463 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.163438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.163642 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.163627 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.168599 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.168579 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n779d\" (UniqueName: \"kubernetes.io/projected/8e36b6d3-0f46-47e0-8cdb-fa279d303b2a-kube-api-access-n779d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv\" (UID: \"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.322291 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.322217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:15.597898 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.597859 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv"] Sep 20 02:04:15.599370 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:04:15.599327 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e36b6d3_0f46_47e0_8cdb_fa279d303b2a.slice/crio-0cc5458f5bd9f22d84eb39d924b55b130f78e6ec35a1617ed535a7cf69e79daa WatchSource:0}: Error finding container 0cc5458f5bd9f22d84eb39d924b55b130f78e6ec35a1617ed535a7cf69e79daa: Status 404 returned error can't find the container with id 0cc5458f5bd9f22d84eb39d924b55b130f78e6ec35a1617ed535a7cf69e79daa Sep 20 02:04:15.600701 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.600685 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 02:04:15.614094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:15.614067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:04:16.514372 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:16.514343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" event={"ID":"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a","Type":"ContainerStarted","Data":"0cc5458f5bd9f22d84eb39d924b55b130f78e6ec35a1617ed535a7cf69e79daa"} Sep 20 02:04:17.521214 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:17.521176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" event={"ID":"8e36b6d3-0f46-47e0-8cdb-fa279d303b2a","Type":"ContainerStarted","Data":"d5113a77b890e4eae1261e006033d271aa018f3b1b5bb98a661dc6128b8cb15c"} Sep 20 02:04:17.537870 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:17.537826 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" podStartSLOduration=2.518341918 podStartE2EDuration="3.537812313s" totalImagesPullingTime="1.019470395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:04:17.536203101 +0000 UTC m=+709.054518869" watchObservedRunningTime="2026-09-20 02:04:17.537812313 +0000 UTC m=+709.056128070" Sep 20 02:04:25.322985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:25.322887 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:25.322985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:25.322966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:25.335959 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:25.335898 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:25.574201 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:25.574117 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv" Sep 20 02:04:27.205446 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:27.205408 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:04:34.622384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:34.622349 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:04:42.107687 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.107625 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr"] Sep 20 02:04:42.111462 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.111440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.113776 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.113755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 20 02:04:42.121575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.121548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr"] Sep 20 02:04:42.205098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12dabe-386b-4154-954b-54c081ce19be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.205275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.205275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.205355 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.205355 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.205355 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.205346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmpt\" (UniqueName: \"kubernetes.io/projected/1c12dabe-386b-4154-954b-54c081ce19be-kube-api-access-wgmpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306184 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12dabe-386b-4154-954b-54c081ce19be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306184 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmpt\" (UniqueName: \"kubernetes.io/projected/1c12dabe-386b-4154-954b-54c081ce19be-kube-api-access-wgmpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306749 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306770 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.306843 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.306813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.308781 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.308751 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c12dabe-386b-4154-954b-54c081ce19be-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.308929 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.308891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c12dabe-386b-4154-954b-54c081ce19be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.314422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.314405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmpt\" (UniqueName: \"kubernetes.io/projected/1c12dabe-386b-4154-954b-54c081ce19be-kube-api-access-wgmpt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gtvzr\" (UID: \"1c12dabe-386b-4154-954b-54c081ce19be\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.424771 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.424729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:42.696855 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:42.696821 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr"] Sep 20 02:04:42.698195 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:04:42.698170 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c12dabe_386b_4154_954b_54c081ce19be.slice/crio-529009c68fe30a3b5b86e50a2ab89629f3be9c1be6aef1a1762eceba86c728e6 WatchSource:0}: Error finding container 529009c68fe30a3b5b86e50a2ab89629f3be9c1be6aef1a1762eceba86c728e6: Status 404 returned error can't find the container with id 529009c68fe30a3b5b86e50a2ab89629f3be9c1be6aef1a1762eceba86c728e6 Sep 20 02:04:43.662460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:43.662426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" event={"ID":"1c12dabe-386b-4154-954b-54c081ce19be","Type":"ContainerStarted","Data":"fe16435e162fc0df4281f54190a2351dafbefe8073c0e4a70e3ee02a381d18f6"} Sep 20 02:04:43.663002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:43.662480 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" event={"ID":"1c12dabe-386b-4154-954b-54c081ce19be","Type":"ContainerStarted","Data":"529009c68fe30a3b5b86e50a2ab89629f3be9c1be6aef1a1762eceba86c728e6"} Sep 20 02:04:43.684233 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:43.684180 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" podStartSLOduration=1.5245045579999998 podStartE2EDuration="1.684164115s" totalImagesPullingTime="159.659557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:04:43.683059422 +0000 UTC m=+735.201375182" watchObservedRunningTime="2026-09-20 02:04:43.684164115 +0000 UTC m=+735.202479872" Sep 20 02:04:43.709232 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:43.709195 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:04:52.425416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:52.425375 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:52.425884 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:52.425447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:52.437884 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:52.437864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:52.722769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:52.722684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gtvzr" Sep 20 02:04:55.520675 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:04:55.520641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:05:00.914213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:00.914175 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:05:15.911009 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.910975 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz"] Sep 20 02:05:15.914634 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.914611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.916885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.916861 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 20 02:05:15.924693 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.924671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz"] Sep 20 02:05:15.984311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0811b7c1-390b-42b4-88b6-764871f70f8b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.984455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.984455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.984455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.984612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:15.984612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:15.984502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqmr\" (UniqueName: \"kubernetes.io/projected/0811b7c1-390b-42b4-88b6-764871f70f8b-kube-api-access-bfqmr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086007 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.085974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqmr\" (UniqueName: \"kubernetes.io/projected/0811b7c1-390b-42b4-88b6-764871f70f8b-kube-api-access-bfqmr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086366 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0811b7c1-390b-42b4-88b6-764871f70f8b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086421 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086421 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.086529 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.086469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.088570 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.088548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0811b7c1-390b-42b4-88b6-764871f70f8b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.088709 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.088690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0811b7c1-390b-42b4-88b6-764871f70f8b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.093695 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.093671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqmr\" (UniqueName: \"kubernetes.io/projected/0811b7c1-390b-42b4-88b6-764871f70f8b-kube-api-access-bfqmr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz\" (UID: \"0811b7c1-390b-42b4-88b6-764871f70f8b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.224964 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.224881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:16.498002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.497977 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz"] Sep 20 02:05:16.499529 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:05:16.499501 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0811b7c1_390b_42b4_88b6_764871f70f8b.slice/crio-f0a6745c522a1e235a38e9334c2d83fed02c6a8db05d853582f48bb330b30e85 WatchSource:0}: Error finding container f0a6745c522a1e235a38e9334c2d83fed02c6a8db05d853582f48bb330b30e85: Status 404 returned error can't find the container with id f0a6745c522a1e235a38e9334c2d83fed02c6a8db05d853582f48bb330b30e85 Sep 20 02:05:16.845195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.845161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" event={"ID":"0811b7c1-390b-42b4-88b6-764871f70f8b","Type":"ContainerStarted","Data":"14927258ecf6f884a3cce1fb21608d88395243c5bd5215e39384290f57aab0f8"} Sep 20 02:05:16.845195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.845201 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" event={"ID":"0811b7c1-390b-42b4-88b6-764871f70f8b","Type":"ContainerStarted","Data":"f0a6745c522a1e235a38e9334c2d83fed02c6a8db05d853582f48bb330b30e85"} Sep 20 02:05:16.861859 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:16.861811 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" podStartSLOduration=1.6637013330000001 podStartE2EDuration="1.861797754s" totalImagesPullingTime="198.096421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:05:16.859865661 +0000 UTC m=+768.378181420" watchObservedRunningTime="2026-09-20 02:05:16.861797754 +0000 UTC m=+768.380113512" Sep 20 02:05:18.311196 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:18.311153 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:05:26.225223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:26.225188 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:26.225790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:26.225242 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:26.237205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:26.237174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:05:26.915326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:05:26.915299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz" Sep 20 02:06:31.272471 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.272435 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:06:31.273039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.272645 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" podUID="f3ca9761-6987-47ee-b812-24fde35ed17e" containerName="authorino" containerID="cri-o://c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922" gracePeriod=30 Sep 20 02:06:31.519504 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.519477 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:06:31.577770 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.577682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2n7l\" (UniqueName: \"kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l\") pod \"f3ca9761-6987-47ee-b812-24fde35ed17e\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " Sep 20 02:06:31.577959 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.577795 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert\") pod \"f3ca9761-6987-47ee-b812-24fde35ed17e\" (UID: \"f3ca9761-6987-47ee-b812-24fde35ed17e\") " Sep 20 02:06:31.580071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.580042 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l" (OuterVolumeSpecName: "kube-api-access-s2n7l") pod "f3ca9761-6987-47ee-b812-24fde35ed17e" (UID: "f3ca9761-6987-47ee-b812-24fde35ed17e"). InnerVolumeSpecName "kube-api-access-s2n7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:06:31.589213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.589183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f3ca9761-6987-47ee-b812-24fde35ed17e" (UID: "f3ca9761-6987-47ee-b812-24fde35ed17e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:06:31.679232 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.679186 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ca9761-6987-47ee-b812-24fde35ed17e-tls-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:06:31.679407 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:31.679249 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2n7l\" (UniqueName: \"kubernetes.io/projected/f3ca9761-6987-47ee-b812-24fde35ed17e-kube-api-access-s2n7l\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:06:32.267184 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.267149 2578 generic.go:356] "Generic (PLEG): container finished" podID="f3ca9761-6987-47ee-b812-24fde35ed17e" containerID="c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922" exitCode=0 Sep 20 02:06:32.267391 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.267216 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" Sep 20 02:06:32.267391 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.267236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" event={"ID":"f3ca9761-6987-47ee-b812-24fde35ed17e","Type":"ContainerDied","Data":"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922"} Sep 20 02:06:32.267391 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.267285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ddfb79c57-bd8r4" event={"ID":"f3ca9761-6987-47ee-b812-24fde35ed17e","Type":"ContainerDied","Data":"b6cb1b2bac1f2b33264db80ffd7b23c90b88506e2e3ce794939eb9e8fc13ceff"} Sep 20 02:06:32.267391 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.267302 2578 scope.go:117] "RemoveContainer" containerID="c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922" Sep 20 02:06:32.276804 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.276660 2578 scope.go:117] "RemoveContainer" containerID="c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922" Sep 20 02:06:32.277037 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:06:32.277005 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922\": container with ID starting with c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922 not found: ID does not exist" containerID="c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922" Sep 20 02:06:32.277081 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.277033 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922"} err="failed to get container status \"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922\": rpc error: code = NotFound desc = could not find container \"c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922\": container with ID starting with c8b047e0efc3f5e5511956a606da18978c0c096713e9720240c1640628d33922 not found: ID does not exist" Sep 20 02:06:32.299628 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.299593 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:06:32.309645 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:32.309617 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ddfb79c57-bd8r4"] Sep 20 02:06:33.017475 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:33.017445 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ca9761-6987-47ee-b812-24fde35ed17e" path="/var/lib/kubelet/pods/f3ca9761-6987-47ee-b812-24fde35ed17e/volumes" Sep 20 02:06:57.161187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.161149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-1b61de"] Sep 20 02:06:57.161844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.161590 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ca9761-6987-47ee-b812-24fde35ed17e" containerName="authorino" Sep 20 02:06:57.161844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.161606 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ca9761-6987-47ee-b812-24fde35ed17e" containerName="authorino" Sep 20 02:06:57.161844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.161677 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ca9761-6987-47ee-b812-24fde35ed17e" containerName="authorino" Sep 20 02:06:57.164703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.164682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:06:57.166978 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.166953 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l75z\"" Sep 20 02:06:57.171866 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.171844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-1b61de"] Sep 20 02:06:57.319384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.319340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knt8h\" (UniqueName: \"kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h\") pod \"test-curl-7543-1b61de\" (UID: \"88cc5c54-462c-47a5-8f7a-46dcb60f4764\") " pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:06:57.420799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.420705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knt8h\" (UniqueName: \"kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h\") pod \"test-curl-7543-1b61de\" (UID: \"88cc5c54-462c-47a5-8f7a-46dcb60f4764\") " pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:06:57.428951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.428897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knt8h\" (UniqueName: \"kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h\") pod \"test-curl-7543-1b61de\" (UID: \"88cc5c54-462c-47a5-8f7a-46dcb60f4764\") " pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:06:57.475077 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.475042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:06:57.748150 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:57.748126 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-1b61de"] Sep 20 02:06:57.750041 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:06:57.750013 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88cc5c54_462c_47a5_8f7a_46dcb60f4764.slice/crio-a6a6e6da8e8dbd1ccf7ab26a88ba3f1e651c4deedb7707f356d17e349d2e4e06 WatchSource:0}: Error finding container a6a6e6da8e8dbd1ccf7ab26a88ba3f1e651c4deedb7707f356d17e349d2e4e06: Status 404 returned error can't find the container with id a6a6e6da8e8dbd1ccf7ab26a88ba3f1e651c4deedb7707f356d17e349d2e4e06 Sep 20 02:06:58.414072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:58.414036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-1b61de" event={"ID":"88cc5c54-462c-47a5-8f7a-46dcb60f4764","Type":"ContainerStarted","Data":"a6a6e6da8e8dbd1ccf7ab26a88ba3f1e651c4deedb7707f356d17e349d2e4e06"} Sep 20 02:06:59.522752 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:06:59.522715 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:07:04.454344 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.454244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-1b61de" event={"ID":"88cc5c54-462c-47a5-8f7a-46dcb60f4764","Type":"ContainerStarted","Data":"ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7"} Sep 20 02:07:04.471999 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.471939 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-1b61de" podStartSLOduration=1.024620516 podStartE2EDuration="7.471921574s" totalImagesPullingTime="6.447301058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:04.469114551 +0000 UTC m=+875.987430309" watchObservedRunningTime="2026-09-20 02:07:04.471921574 +0000 UTC m=+875.990237326" Sep 20 02:07:04.922305 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.922270 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:04.926358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.926332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.929007 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.928979 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 20 02:07:04.929152 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.928980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c4557f0d-openshift-default-dockercfg-hl268\"" Sep 20 02:07:04.939549 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.939508 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:04.978157 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9flt\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:04.978572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:04.978519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9flt\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.079955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.079840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.080312 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.080153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.080371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.080310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.080371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.080360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.080473 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.080427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.080602 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.080585 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.082548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.082524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.082762 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.082746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.087542 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.087522 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.087869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.087848 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9flt\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt\") pod \"e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.119587 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.119550 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-1b61de"] Sep 20 02:07:05.123069 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.123030 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-1b61de"] Sep 20 02:07:05.241181 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.241149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:05.261171 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.261144 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-ddd460"] Sep 20 02:07:05.261568 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.261554 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" containerName="test-curl-7543-1b61de" Sep 20 02:07:05.261616 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.261571 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" containerName="test-curl-7543-1b61de" Sep 20 02:07:05.261652 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.261646 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" containerName="test-curl-7543-1b61de" Sep 20 02:07:05.265091 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.265065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:05.268604 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.268573 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-ddd460"] Sep 20 02:07:05.282173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.282145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwm9\" (UniqueName: \"kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9\") pod \"test-curl-7543-ddd460\" (UID: \"cb760f94-ef09-4ac8-94d2-81271bfc1cbc\") " pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:05.382874 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.382837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwm9\" (UniqueName: \"kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9\") pod \"test-curl-7543-ddd460\" (UID: \"cb760f94-ef09-4ac8-94d2-81271bfc1cbc\") " pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:05.390853 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.390831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwm9\" (UniqueName: \"kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9\") pod \"test-curl-7543-ddd460\" (UID: \"cb760f94-ef09-4ac8-94d2-81271bfc1cbc\") " pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:05.459977 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:05.459945 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-1b61de" container="test-curl-7543-1b61de" Sep 20 02:07:05.520579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.520549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:05.522073 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:05.522043 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e65ac6f_0bec_4d0c_82ca_4e1914641704.slice/crio-00e201a27cac86b6b0aab5580929f37e1bf235b8a203caa6c18cb89a116e22c7 WatchSource:0}: Error finding container 00e201a27cac86b6b0aab5580929f37e1bf235b8a203caa6c18cb89a116e22c7: Status 404 returned error can't find the container with id 00e201a27cac86b6b0aab5580929f37e1bf235b8a203caa6c18cb89a116e22c7 Sep 20 02:07:05.579660 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.579629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:05.851540 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:05.851516 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-ddd460"] Sep 20 02:07:05.853686 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:05.853665 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb760f94_ef09_4ac8_94d2_81271bfc1cbc.slice/crio-627fc8188d055b043dfa98a7166db2dfd981ca2bf4d9b9bd8bca3de46208515f WatchSource:0}: Error finding container 627fc8188d055b043dfa98a7166db2dfd981ca2bf4d9b9bd8bca3de46208515f: Status 404 returned error can't find the container with id 627fc8188d055b043dfa98a7166db2dfd981ca2bf4d9b9bd8bca3de46208515f Sep 20 02:07:06.467942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:06.467889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-ddd460" event={"ID":"cb760f94-ef09-4ac8-94d2-81271bfc1cbc","Type":"ContainerStarted","Data":"0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912"} Sep 20 02:07:06.468403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:06.467953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-ddd460" event={"ID":"cb760f94-ef09-4ac8-94d2-81271bfc1cbc","Type":"ContainerStarted","Data":"627fc8188d055b043dfa98a7166db2dfd981ca2bf4d9b9bd8bca3de46208515f"} Sep 20 02:07:06.469315 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:06.469290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerStarted","Data":"00e201a27cac86b6b0aab5580929f37e1bf235b8a203caa6c18cb89a116e22c7"} Sep 20 02:07:06.469419 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:06.469364 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-1b61de" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" containerName="test-curl-7543-1b61de" containerID="cri-o://ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7" gracePeriod=2 Sep 20 02:07:06.487137 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:06.487085 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7543-1b61de\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:07:06.488492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:06.488434 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-ddd460" podStartSLOduration=1.488412627 podStartE2EDuration="1.488412627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:06.484861008 +0000 UTC m=+878.003176767" watchObservedRunningTime="2026-09-20 02:07:06.488412627 +0000 UTC m=+878.006728386" Sep 20 02:07:07.142942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:07.142826 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-ddd460"] Sep 20 02:07:07.161928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:07.161868 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-ddd460"] Sep 20 02:07:07.475327 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:07.475255 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-ddd460" container="test-curl-7543-ddd460" Sep 20 02:07:08.022285 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.022240 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:07:08.022940 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.022920 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" containerName="test-curl-7543-ddd460" Sep 20 02:07:08.022996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.022944 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" containerName="test-curl-7543-ddd460" Sep 20 02:07:08.023076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.023066 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" containerName="test-curl-7543-ddd460" Sep 20 02:07:08.026729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.026707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.029119 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.029097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4cfa5a-openshift-default-dockercfg-pzd66\"" Sep 20 02:07:08.038666 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.038640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:07:08.108136 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.108570 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.108504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bfq\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209566 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209566 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209566 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209566 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209566 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bfq\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209826 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209826 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209826 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209881 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.209994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.209972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.210165 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.210145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.210393 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.210371 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.211898 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.211878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.212067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.212045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.218192 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.218168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.218464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.218442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bfq\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq\") pod \"e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.341703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.341593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:08.489065 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.489018 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-ddd460" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" containerName="test-curl-7543-ddd460" containerID="cri-o://0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912" gracePeriod=2 Sep 20 02:07:08.491348 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:08.491317 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7543-ddd460\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:08.617397 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.617368 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:07:08.619509 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:08.619475 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7543-ddd460\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:08.627004 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.626976 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:07:08.628270 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:08.628237 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905e594d_f592_4d39_a953_15563efc240a.slice/crio-9ad1e85b914e7e690df810e207858a0b88952b50ffbd85eedd48e40a2d0948ef WatchSource:0}: Error finding container 9ad1e85b914e7e690df810e207858a0b88952b50ffbd85eedd48e40a2d0948ef: Status 404 returned error can't find the container with id 9ad1e85b914e7e690df810e207858a0b88952b50ffbd85eedd48e40a2d0948ef Sep 20 02:07:08.714824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.714787 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knt8h\" (UniqueName: \"kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h\") pod \"88cc5c54-462c-47a5-8f7a-46dcb60f4764\" (UID: \"88cc5c54-462c-47a5-8f7a-46dcb60f4764\") " Sep 20 02:07:08.717215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.717183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h" (OuterVolumeSpecName: "kube-api-access-knt8h") pod "88cc5c54-462c-47a5-8f7a-46dcb60f4764" (UID: "88cc5c54-462c-47a5-8f7a-46dcb60f4764"). InnerVolumeSpecName "kube-api-access-knt8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:07:08.816179 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:08.816141 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knt8h\" (UniqueName: \"kubernetes.io/projected/88cc5c54-462c-47a5-8f7a-46dcb60f4764-kube-api-access-knt8h\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:09.016540 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:09.016496 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7543-ddd460\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:09.017171 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.017149 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" path="/var/lib/kubelet/pods/88cc5c54-462c-47a5-8f7a-46dcb60f4764/volumes" Sep 20 02:07:09.498812 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.498768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerStarted","Data":"9ad1e85b914e7e690df810e207858a0b88952b50ffbd85eedd48e40a2d0948ef"} Sep 20 02:07:09.500457 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.500421 2578 generic.go:356] "Generic (PLEG): container finished" podID="88cc5c54-462c-47a5-8f7a-46dcb60f4764" containerID="ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7" exitCode=137 Sep 20 02:07:09.500595 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.500467 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-1b61de" Sep 20 02:07:09.500595 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.500540 2578 scope.go:117] "RemoveContainer" containerID="ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7" Sep 20 02:07:09.695952 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.695928 2578 scope.go:117] "RemoveContainer" containerID="ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7" Sep 20 02:07:09.696342 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:09.696323 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7\": container with ID starting with ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7 not found: ID does not exist" containerID="ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7" Sep 20 02:07:09.696436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.696353 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7"} err="failed to get container status \"ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7\": rpc error: code = NotFound desc = could not find container \"ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7\": container with ID starting with ce3c99ec321103e8e81efd47791e4670a1373c90a4cea8033acf6e8501d536b7 not found: ID does not exist" Sep 20 02:07:09.776227 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776161 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:09.776326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776271 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:09.776326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776312 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:09.776665 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776638 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:09.776725 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776711 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:09.776783 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:09.776751 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:10.508351 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.508316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerStarted","Data":"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c"} Sep 20 02:07:10.509970 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.509940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerStarted","Data":"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062"} Sep 20 02:07:10.529796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.529737 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podStartSLOduration=2.278639727 podStartE2EDuration="6.529715212s" totalImagesPullingTime="4.251075485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:10.526240978 +0000 UTC m=+882.044556736" watchObservedRunningTime="2026-09-20 02:07:10.529715212 +0000 UTC m=+882.048030972" Sep 20 02:07:10.544482 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.544432 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podStartSLOduration=1.398948925 podStartE2EDuration="2.544413959s" totalImagesPullingTime="1.145465034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:10.543090436 +0000 UTC m=+882.061406196" watchObservedRunningTime="2026-09-20 02:07:10.544413959 +0000 UTC m=+882.062729719" Sep 20 02:07:10.666545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.666513 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:10.735782 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.735745 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cwm9\" (UniqueName: \"kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9\") pod \"cb760f94-ef09-4ac8-94d2-81271bfc1cbc\" (UID: \"cb760f94-ef09-4ac8-94d2-81271bfc1cbc\") " Sep 20 02:07:10.738056 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.738025 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9" (OuterVolumeSpecName: "kube-api-access-9cwm9") pod "cb760f94-ef09-4ac8-94d2-81271bfc1cbc" (UID: "cb760f94-ef09-4ac8-94d2-81271bfc1cbc"). InnerVolumeSpecName "kube-api-access-9cwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:07:10.836951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:10.836891 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwm9\" (UniqueName: \"kubernetes.io/projected/cb760f94-ef09-4ac8-94d2-81271bfc1cbc-kube-api-access-9cwm9\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:11.016092 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.015993 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" path="/var/lib/kubelet/pods/cb760f94-ef09-4ac8-94d2-81271bfc1cbc/volumes" Sep 20 02:07:11.241327 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.241282 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:11.246301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.246274 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:11.342078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.341990 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:11.347243 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.347217 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:11.518167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.518134 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb760f94-ef09-4ac8-94d2-81271bfc1cbc" containerID="0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912" exitCode=137 Sep 20 02:07:11.518167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.518181 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-ddd460" Sep 20 02:07:11.518719 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.518233 2578 scope.go:117] "RemoveContainer" containerID="0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912" Sep 20 02:07:11.518976 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.518823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:11.518976 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.518965 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:11.519875 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.519854 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:11.520042 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.519966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:11.528054 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.527860 2578 scope.go:117] "RemoveContainer" containerID="0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912" Sep 20 02:07:11.528180 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:11.528160 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912\": container with ID starting with 0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912 not found: ID does not exist" containerID="0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912" Sep 20 02:07:11.528250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:11.528187 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912"} err="failed to get container status \"0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912\": rpc error: code = NotFound desc = could not find container \"0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912\": container with ID starting with 0a6f4a9d038e768c952f111a5782a243603e7040aa5bd667c5957ccf86cc6912 not found: ID does not exist" Sep 20 02:07:15.406995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.406944 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:07:15.413131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.413102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.417005 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.416599 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 20 02:07:15.417005 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.416937 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 20 02:07:15.417232 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.417214 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 20 02:07:15.417439 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.417420 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4cfa5a-serving-cert\"" Sep 20 02:07:15.417774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.417757 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z4spz\"" Sep 20 02:07:15.418359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.418334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:07:15.483844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.483813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.484013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.483950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.484013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.483996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.484117 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.484052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.585159 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.585115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.585316 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.585251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.585316 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.585292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.585455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.585319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.585455 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:15.585450 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-serving-cert: secret "maas-api-e2e-autores-4cfa5a-serving-cert" not found Sep 20 02:07:15.585545 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:15.585528 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls podName:f2648736-4025-4bd2-b349-7c268d5efa54 nodeName:}" failed. No retries permitted until 2026-09-20 02:07:16.085504931 +0000 UTC m=+887.603820670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls") pod "maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" (UID: "f2648736-4025-4bd2-b349-7c268d5efa54") : secret "maas-api-e2e-autores-4cfa5a-serving-cert" not found Sep 20 02:07:15.586012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.585973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.587995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.587965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.593896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.593869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:15.822603 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:15.822508 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:07:16.089723 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.089624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:16.092434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.092410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") pod \"maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:16.348933 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.348829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:16.439503 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.439452 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:16.445276 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.445252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.448936 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.448603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c4557f0d-serving-cert\"" Sep 20 02:07:16.452585 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.452519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:16.494450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.494414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.494614 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.494484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7k96\" (UniqueName: \"kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.494614 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.494541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.494614 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.494564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.595613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.595575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.595613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.595618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.595888 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.595675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.595888 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.595737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7k96\" (UniqueName: \"kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.595888 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:16.595746 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-serving-cert: secret "maas-api-e2e-ait-c4557f0d-serving-cert" not found Sep 20 02:07:16.595888 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:16.595841 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls podName:84ef18ad-688d-4ff8-b26f-949c8f69a8b9 nodeName:}" failed. No retries permitted until 2026-09-20 02:07:17.095819386 +0000 UTC m=+888.614135123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls") pod "maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" (UID: "84ef18ad-688d-4ff8-b26f-949c8f69a8b9") : secret "maas-api-e2e-ait-c4557f0d-serving-cert" not found Sep 20 02:07:16.596474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.596451 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.598352 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.598326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.606173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.606076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7k96\" (UniqueName: \"kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:16.626511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:16.626479 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:07:16.628148 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:16.628120 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2648736_4025_4bd2_b349_7c268d5efa54.slice/crio-57e116ab1d7622bfa22d7d1fb16522877baef8d8ff0064681d5b1cd8a332d077 WatchSource:0}: Error finding container 57e116ab1d7622bfa22d7d1fb16522877baef8d8ff0064681d5b1cd8a332d077: Status 404 returned error can't find the container with id 57e116ab1d7622bfa22d7d1fb16522877baef8d8ff0064681d5b1cd8a332d077 Sep 20 02:07:17.102132 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.102084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:17.104894 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.104863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") pod \"maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:17.360139 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.359798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:17.561242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.561177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" event={"ID":"f2648736-4025-4bd2-b349-7c268d5efa54","Type":"ContainerStarted","Data":"57e116ab1d7622bfa22d7d1fb16522877baef8d8ff0064681d5b1cd8a332d077"} Sep 20 02:07:17.666806 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.666763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:17.669242 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:17.669209 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ef18ad_688d_4ff8_b26f_949c8f69a8b9.slice/crio-9423877a015c62dd3593e1241f5cb0054692eeec147c6b62c7708b35750508f3 WatchSource:0}: Error finding container 9423877a015c62dd3593e1241f5cb0054692eeec147c6b62c7708b35750508f3: Status 404 returned error can't find the container with id 9423877a015c62dd3593e1241f5cb0054692eeec147c6b62c7708b35750508f3 Sep 20 02:07:17.822086 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.822051 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:07:17.827609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.827585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.832502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.832472 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-703998-openshift-default-dockercfg-rgj6f\"" Sep 20 02:07:17.842891 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.842838 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:07:17.911055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911257 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911257 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911257 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911257 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjrq\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911527 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:17.911578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:17.911540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.012944 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.012790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.012944 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.012864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.012944 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.012922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.012960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.012993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjrq\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013588 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.013588 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.013563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.014167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.014133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.014287 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.014246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.016345 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.016283 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.017678 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.017639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.023954 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.023895 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.031381 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.031349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjrq\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq\") pod \"e2e-inf-b-703998-openshift-default-78669cb68d-f479k\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.143425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.143384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:18.574024 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:18.573987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" event={"ID":"84ef18ad-688d-4ff8-b26f-949c8f69a8b9","Type":"ContainerStarted","Data":"9423877a015c62dd3593e1241f5cb0054692eeec147c6b62c7708b35750508f3"} Sep 20 02:07:19.582141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.582044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" event={"ID":"84ef18ad-688d-4ff8-b26f-949c8f69a8b9","Type":"ContainerStarted","Data":"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422"} Sep 20 02:07:19.582599 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.582225 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:19.583846 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.583823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" event={"ID":"f2648736-4025-4bd2-b349-7c268d5efa54","Type":"ContainerStarted","Data":"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd"} Sep 20 02:07:19.583963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.583871 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:19.596552 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.596531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:07:19.598211 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:19.598184 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b6404c_4d47_42ab_9391_b665986090c5.slice/crio-f8c6528986828f29e16c4eeb57cac3a81b64736a8ff5f316c69a08c359556f7c WatchSource:0}: Error finding container f8c6528986828f29e16c4eeb57cac3a81b64736a8ff5f316c69a08c359556f7c: Status 404 returned error can't find the container with id f8c6528986828f29e16c4eeb57cac3a81b64736a8ff5f316c69a08c359556f7c Sep 20 02:07:19.602172 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.602139 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:19.602294 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.602207 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:19.602294 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.602242 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:19.602411 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.602189 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" podStartSLOduration=1.874780465 podStartE2EDuration="3.60217547s" totalImagesPullingTime="1.727395005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:19.600775137 +0000 UTC m=+891.119090908" watchObservedRunningTime="2026-09-20 02:07:19.60217547 +0000 UTC m=+891.120491227" Sep 20 02:07:19.624775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:19.624725 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" podStartSLOduration=1.8669496300000001 podStartE2EDuration="4.624708777s" totalImagesPullingTime="2.757759147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:19.62223083 +0000 UTC m=+891.140546600" watchObservedRunningTime="2026-09-20 02:07:19.624708777 +0000 UTC m=+891.143024569" Sep 20 02:07:20.594990 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:20.594871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerStarted","Data":"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827"} Sep 20 02:07:20.594990 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:20.594974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerStarted","Data":"f8c6528986828f29e16c4eeb57cac3a81b64736a8ff5f316c69a08c359556f7c"} Sep 20 02:07:20.618470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:20.618417 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podStartSLOduration=3.618401347 podStartE2EDuration="3.618401347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:20.615402983 +0000 UTC m=+892.133718742" watchObservedRunningTime="2026-09-20 02:07:20.618401347 +0000 UTC m=+892.136717104" Sep 20 02:07:21.144444 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:21.144400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:21.149629 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:21.149604 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:21.601721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:21.601630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:21.602728 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:21.602705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:25.600276 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:25.600247 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:07:25.600735 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:25.600720 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:29.024522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:29.024490 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:07:29.029151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:29.029125 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:07:37.322593 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.322507 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:07:37.744606 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.744525 2578 generic.go:356] "Generic (PLEG): container finished" podID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerID="4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c" exitCode=0 Sep 20 02:07:37.744812 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.744606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerDied","Data":"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c"} Sep 20 02:07:37.745104 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.745076 2578 scope.go:117] "RemoveContainer" containerID="4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c" Sep 20 02:07:37.746486 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746444 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:37.746577 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746518 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:37.746577 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746555 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:37.746577 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746557 2578 generic.go:356] "Generic (PLEG): container finished" podID="905e594d-f592-4d39-a953-15563efc240a" containerID="bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062" exitCode=0 Sep 20 02:07:37.746698 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerDied","Data":"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062"} Sep 20 02:07:37.746984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.746970 2578 scope.go:117] "RemoveContainer" containerID="bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062" Sep 20 02:07:37.748120 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.748092 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:37.748223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.748167 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:37.748223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:37.748216 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:38.762397 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:38.762354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerStarted","Data":"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7"} Sep 20 02:07:38.767146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:38.767074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerStarted","Data":"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a"} Sep 20 02:07:39.253935 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.241457 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:39.253935 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.247042 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 20 02:07:39.253935 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.247155 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 20 02:07:39.342082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.342032 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:39.343661 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.343627 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:39.343869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:39.343844 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:40.241575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:40.241526 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 20 02:07:40.242139 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:40.241601 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 20 02:07:40.342676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:40.342638 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:40.342966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:40.342938 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:41.241865 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.241822 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 20 02:07:41.242507 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.242473 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 20 02:07:41.343103 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.343054 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:41.343449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.343385 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:41.518643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.518560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:41.519491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:41.519473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:42.241975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:42.241937 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 20 02:07:42.242591 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:42.242559 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 20 02:07:42.350108 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:42.350062 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:42.350425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:42.350392 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:43.241794 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:43.241742 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 20 02:07:43.242050 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:43.241835 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 20 02:07:43.342846 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:43.342796 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:43.343084 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:43.342871 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:44.308087 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:44.308052 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:44.310408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:44.310372 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:44.342449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:44.342410 2578 patch_prober.go:28] interesting pod/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 20 02:07:44.342780 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:44.342746 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 20 02:07:45.348250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:45.348223 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:45.349420 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:45.349399 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:07:46.904285 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.904254 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:07:46.907978 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.907956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.911644 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.911603 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"kube-root-ca.crt\"" Sep 20 02:07:46.911644 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.911629 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"openshift-service-ca.crt\"" Sep 20 02:07:46.911874 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.911675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"simulator-w0-8195ba-kserve-self-signed-certs\"" Sep 20 02:07:46.911874 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.911603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"default-dockercfg-s9bwr\"" Sep 20 02:07:46.916455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.916419 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:07:46.947847 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.947793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.947847 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.947853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.948156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.947955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.948156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.947999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7clh\" (UniqueName: \"kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.948156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.948069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:46.948156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:46.948097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7clh\" (UniqueName: \"kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049728 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.049728 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.049709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.050143 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.050121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.052096 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.052073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.052481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.052458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.057464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.057438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7clh\" (UniqueName: \"kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh\") pod \"simulator-w0-8195ba-kserve-649dd587c-qj7hk\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.221150 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.221047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:47.495895 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.495867 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:07:47.498160 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:47.498134 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf1355d_d8aa_4123_b5bc_25a9ff3ca87e.slice/crio-882a74c17d58af049c8b6f7113988d1b3d9a19b7d332dc6c984732e32abe6618 WatchSource:0}: Error finding container 882a74c17d58af049c8b6f7113988d1b3d9a19b7d332dc6c984732e32abe6618: Status 404 returned error can't find the container with id 882a74c17d58af049c8b6f7113988d1b3d9a19b7d332dc6c984732e32abe6618 Sep 20 02:07:47.844163 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.844077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" event={"ID":"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e","Type":"ContainerStarted","Data":"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea"} Sep 20 02:07:47.844163 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.844118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" event={"ID":"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e","Type":"ContainerStarted","Data":"882a74c17d58af049c8b6f7113988d1b3d9a19b7d332dc6c984732e32abe6618"} Sep 20 02:07:47.862379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:47.862317 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" podStartSLOduration=1.862299212 podStartE2EDuration="1.862299212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:47.859085882 +0000 UTC m=+919.377401675" watchObservedRunningTime="2026-09-20 02:07:47.862299212 +0000 UTC m=+919.380614969" Sep 20 02:07:48.851070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.851039 2578 generic.go:356] "Generic (PLEG): container finished" podID="59b6404c-4d47-42ab-9391-b665986090c5" containerID="681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827" exitCode=0 Sep 20 02:07:48.851487 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.851112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerDied","Data":"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827"} Sep 20 02:07:48.851578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.851564 2578 scope.go:117] "RemoveContainer" containerID="681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827" Sep 20 02:07:48.852713 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.852686 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:48.852792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.852761 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:48.852828 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:48.852794 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:49.862273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:49.862236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerStarted","Data":"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86"} Sep 20 02:07:50.144815 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:50.144771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:50.146424 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:50.146379 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:50.146623 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:50.146456 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:51.144478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:51.144427 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:51.145026 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:51.144503 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:51.602117 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:51.602010 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:52.144422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:52.144374 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:52.144675 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:52.144454 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:53.144486 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:53.144443 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:53.144793 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:53.144759 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:54.009363 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.009324 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:54.009694 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.009656 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" podUID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" containerName="maas-api" containerID="cri-o://6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422" gracePeriod=30 Sep 20 02:07:54.147169 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.145354 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:54.147169 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.145444 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:54.307491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.307451 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:54.371129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.370113 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:54.371129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.370432 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" containerID="cri-o://0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a" gracePeriod=30 Sep 20 02:07:54.428622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.428565 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca\") pod \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " Sep 20 02:07:54.428820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.428690 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") pod \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " Sep 20 02:07:54.428820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.428722 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7k96\" (UniqueName: \"kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96\") pod \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " Sep 20 02:07:54.428820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.428758 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs\") pod \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\" (UID: \"84ef18ad-688d-4ff8-b26f-949c8f69a8b9\") " Sep 20 02:07:54.429048 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.428948 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84ef18ad-688d-4ff8-b26f-949c8f69a8b9" (UID: "84ef18ad-688d-4ff8-b26f-949c8f69a8b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:07:54.429161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.429137 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:54.431375 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.431341 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84ef18ad-688d-4ff8-b26f-949c8f69a8b9" (UID: "84ef18ad-688d-4ff8-b26f-949c8f69a8b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:07:54.431548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.431527 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84ef18ad-688d-4ff8-b26f-949c8f69a8b9" (UID: "84ef18ad-688d-4ff8-b26f-949c8f69a8b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:07:54.431702 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.431684 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96" (OuterVolumeSpecName: "kube-api-access-b7k96") pod "84ef18ad-688d-4ff8-b26f-949c8f69a8b9" (UID: "84ef18ad-688d-4ff8-b26f-949c8f69a8b9"). InnerVolumeSpecName "kube-api-access-b7k96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:07:54.530100 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.530061 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:54.530100 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.530094 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7k96\" (UniqueName: \"kubernetes.io/projected/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-kube-api-access-b7k96\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:54.530100 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.530108 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ef18ad-688d-4ff8-b26f-949c8f69a8b9-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:54.727762 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.727713 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:07:54.899051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.896951 2578 generic.go:356] "Generic (PLEG): container finished" podID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" containerID="6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422" exitCode=0 Sep 20 02:07:54.899051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.897108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" event={"ID":"84ef18ad-688d-4ff8-b26f-949c8f69a8b9","Type":"ContainerDied","Data":"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422"} Sep 20 02:07:54.899051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.897163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" event={"ID":"84ef18ad-688d-4ff8-b26f-949c8f69a8b9","Type":"ContainerDied","Data":"9423877a015c62dd3593e1241f5cb0054692eeec147c6b62c7708b35750508f3"} Sep 20 02:07:54.899051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.897195 2578 scope.go:117] "RemoveContainer" containerID="6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422" Sep 20 02:07:54.899051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.897374 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228" Sep 20 02:07:54.911627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.911604 2578 scope.go:117] "RemoveContainer" containerID="6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422" Sep 20 02:07:54.911991 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:54.911969 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422\": container with ID starting with 6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422 not found: ID does not exist" containerID="6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422" Sep 20 02:07:54.912044 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.912002 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422"} err="failed to get container status \"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422\": rpc error: code = NotFound desc = could not find container \"6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422\": container with ID starting with 6ffdb23fb1cfed969385e31206399d2bf9692507c41cf05093475ced0e490422 not found: ID does not exist" Sep 20 02:07:54.922059 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.922027 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:54.924705 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:54.924672 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4557f0d-7df54b64c8-q2228"] Sep 20 02:07:55.016024 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.015939 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" path="/var/lib/kubelet/pods/84ef18ad-688d-4ff8-b26f-949c8f69a8b9/volumes" Sep 20 02:07:55.144461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.144421 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 20 02:07:55.144631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.144482 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 20 02:07:55.429640 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.429601 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:07:55.430273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.430255 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" containerName="maas-api" Sep 20 02:07:55.430273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.430275 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" containerName="maas-api" Sep 20 02:07:55.430416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.430378 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ef18ad-688d-4ff8-b26f-949c8f69a8b9" containerName="maas-api" Sep 20 02:07:55.433737 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.433716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.437191 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.437167 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-34c720e3-gw-openshift-default-dockercfg-xnhfp\"" Sep 20 02:07:55.448264 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.448236 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:07:55.538438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjd7s\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.538966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.538803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.639962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.639955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjd7s\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640499 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.640559 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.640540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.641073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.641049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.641073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.641062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.642564 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.642540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.642871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.642849 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.653051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.653020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjd7s\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.661265 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.661232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token\") pod \"e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:55.747942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:55.747825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:56.029246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.029209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:07:56.030038 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:07:56.030014 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5cc6b3_327b_48bf_89dd_dc40852fb92f.slice/crio-d47916bf1c326ad680224df9d10fe3810a77789abc3b2cc7ff3bdf6a55a60948 WatchSource:0}: Error finding container d47916bf1c326ad680224df9d10fe3810a77789abc3b2cc7ff3bdf6a55a60948: Status 404 returned error can't find the container with id d47916bf1c326ad680224df9d10fe3810a77789abc3b2cc7ff3bdf6a55a60948 Sep 20 02:07:56.032571 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.032537 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:56.032663 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.032609 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:56.032663 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.032648 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:07:56.149766 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.149732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:56.151071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.151047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:07:56.918708 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.918671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerStarted","Data":"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c"} Sep 20 02:07:56.919190 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.918728 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerStarted","Data":"d47916bf1c326ad680224df9d10fe3810a77789abc3b2cc7ff3bdf6a55a60948"} Sep 20 02:07:56.941519 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:56.941461 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podStartSLOduration=1.941442709 podStartE2EDuration="1.941442709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:07:56.938791514 +0000 UTC m=+928.457107264" watchObservedRunningTime="2026-09-20 02:07:56.941442709 +0000 UTC m=+928.459758467" Sep 20 02:07:57.221868 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.221767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:57.221868 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.221828 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:57.234733 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.234702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:57.748396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.748362 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:57.753747 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.753718 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:57.924576 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.924540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:57.925646 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.925626 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:07:57.936360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:57.936333 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:07:59.312005 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.311964 2578 patch_prober.go:28] interesting pod/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 20 02:07:59.312427 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.312030 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 20 02:07:59.416408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.416369 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:07:59.419800 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.419773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.423467 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.423439 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-34c720e3-serving-cert\"" Sep 20 02:07:59.431949 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.431843 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:07:59.482773 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.482729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.484306 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.484267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.484750 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.484731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.484867 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.484852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kl6w\" (UniqueName: \"kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.586173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.586074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kl6w\" (UniqueName: \"kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.586173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.586132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.586173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.586171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.586437 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.586281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.586437 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:59.586321 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-serving-cert: secret "maas-api-e2e-migrate-34c720e3-serving-cert" not found Sep 20 02:07:59.586437 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:59.586402 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls podName:7cef3897-0f09-4b7a-830a-227ccd405138 nodeName:}" failed. No retries permitted until 2026-09-20 02:08:00.086379354 +0000 UTC m=+931.604695103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls") pod "maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" (UID: "7cef3897-0f09-4b7a-830a-227ccd405138") : secret "maas-api-e2e-migrate-34c720e3-serving-cert" not found Sep 20 02:07:59.586960 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.586936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.588987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.588927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.595073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.595053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kl6w\" (UniqueName: \"kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:07:59.731929 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.731884 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:59.889496 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889463 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889503 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889549 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9flt\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889593 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889628 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889655 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889711 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889746 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token\") pod \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\" (UID: \"4e65ac6f-0bec-4d0c-82ca-4e1914641704\") " Sep 20 02:07:59.889983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.889888 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:07:59.890139 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890053 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:07:59.890201 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890157 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:07:59.890337 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890305 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:07:59.890337 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890318 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:07:59.890702 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890383 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.890702 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890401 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.890702 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.890410 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.892286 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.892261 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:07:59.892404 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.892378 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt" (OuterVolumeSpecName: "kube-api-access-z9flt") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "kube-api-access-z9flt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:07:59.892472 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.892451 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:07:59.892514 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.892492 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e65ac6f-0bec-4d0c-82ca-4e1914641704" (UID: "4e65ac6f-0bec-4d0c-82ca-4e1914641704"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:07:59.937637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.937599 2578 generic.go:356] "Generic (PLEG): container finished" podID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerID="0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a" exitCode=0 Sep 20 02:07:59.937845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.937644 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" Sep 20 02:07:59.937845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.937674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerDied","Data":"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a"} Sep 20 02:07:59.937845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.937760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc" event={"ID":"4e65ac6f-0bec-4d0c-82ca-4e1914641704","Type":"ContainerDied","Data":"00e201a27cac86b6b0aab5580929f37e1bf235b8a203caa6c18cb89a116e22c7"} Sep 20 02:07:59.937845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.937784 2578 scope.go:117] "RemoveContainer" containerID="0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a" Sep 20 02:07:59.947837 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.947798 2578 scope.go:117] "RemoveContainer" containerID="4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c" Sep 20 02:07:59.956866 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.956846 2578 scope.go:117] "RemoveContainer" containerID="0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a" Sep 20 02:07:59.957174 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:59.957157 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a\": container with ID starting with 0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a not found: ID does not exist" containerID="0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a" Sep 20 02:07:59.957223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.957184 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a"} err="failed to get container status \"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a\": rpc error: code = NotFound desc = could not find container \"0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a\": container with ID starting with 0cfba6dd9ce3b60590f1c507f77e35c113592219651edd4f1487790f2174fc5a not found: ID does not exist" Sep 20 02:07:59.957223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.957201 2578 scope.go:117] "RemoveContainer" containerID="4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c" Sep 20 02:07:59.957451 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:07:59.957433 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c\": container with ID starting with 4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c not found: ID does not exist" containerID="4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c" Sep 20 02:07:59.957498 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.957458 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c"} err="failed to get container status \"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c\": rpc error: code = NotFound desc = could not find container \"4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c\": container with ID starting with 4bddf49d962c3e003d0c4deeaed749b5468e4cd8ce78349310c2964b4060e71c not found: ID does not exist" Sep 20 02:07:59.975985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.975919 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:59.979791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.979765 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c4557f0d-openshift-default-76d4f8cd8d-2gbgc"] Sep 20 02:07:59.991559 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991526 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.991559 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991553 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.991559 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991562 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.991772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991570 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9flt\" (UniqueName: \"kubernetes.io/projected/4e65ac6f-0bec-4d0c-82ca-4e1914641704-kube-api-access-z9flt\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.991772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991581 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:07:59.991772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:07:59.991590 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e65ac6f-0bec-4d0c-82ca-4e1914641704-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:00.092775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:00.092728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:00.095245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:00.095220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") pod \"maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:00.335102 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:00.335003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:00.609553 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:00.609524 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:08:00.611337 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:08:00.611305 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cef3897_0f09_4b7a_830a_227ccd405138.slice/crio-f01bba01e2555c5bf83e3949911b6cdb336365fa2e4d3bc8f2dd1853642588c7 WatchSource:0}: Error finding container f01bba01e2555c5bf83e3949911b6cdb336365fa2e4d3bc8f2dd1853642588c7: Status 404 returned error can't find the container with id f01bba01e2555c5bf83e3949911b6cdb336365fa2e4d3bc8f2dd1853642588c7 Sep 20 02:08:00.952049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:00.952017 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" event={"ID":"7cef3897-0f09-4b7a-830a-227ccd405138","Type":"ContainerStarted","Data":"f01bba01e2555c5bf83e3949911b6cdb336365fa2e4d3bc8f2dd1853642588c7"} Sep 20 02:08:01.017072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:01.017040 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" path="/var/lib/kubelet/pods/4e65ac6f-0bec-4d0c-82ca-4e1914641704/volumes" Sep 20 02:08:01.959807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:01.959770 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" event={"ID":"7cef3897-0f09-4b7a-830a-227ccd405138","Type":"ContainerStarted","Data":"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24"} Sep 20 02:08:01.960272 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:01.959920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:01.978285 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:01.978226 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" podStartSLOduration=2.663337716 podStartE2EDuration="2.978209155s" totalImagesPullingTime="314.871439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:08:01.976046541 +0000 UTC m=+933.494362298" watchObservedRunningTime="2026-09-20 02:08:01.978209155 +0000 UTC m=+933.496524912" Sep 20 02:08:07.973386 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:07.973353 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:08.098986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.098950 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:08:08.099397 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.099382 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:08.099449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.099399 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:08.099492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.099472 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:08.099492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.099482 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:08.102020 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.101998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.104271 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.104235 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b15051\"/\"kube-root-ca.crt\"" Sep 20 02:08:08.104423 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.104294 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b15051\"/\"openshift-service-ca.crt\"" Sep 20 02:08:08.104495 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.104404 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b15051\"/\"default-dockercfg-xgpxq\"" Sep 20 02:08:08.104549 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.104508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b15051\"/\"simulator-w4-b15051-kserve-self-signed-certs\"" Sep 20 02:08:08.109667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.109642 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:08:08.272993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.272855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.272993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.272901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zp4\" (UniqueName: \"kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.272993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.272964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.273226 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.273060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.273226 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.273095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.273226 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.273129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374377 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374377 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374377 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zp4\" (UniqueName: \"kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374377 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374606 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374806 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374750 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.374806 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.374770 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.375145 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.375119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.377359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.377337 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.377790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.377745 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.384603 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.384566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zp4\" (UniqueName: \"kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4\") pod \"simulator-w4-b15051-kserve-7df9778df9-pdjdr\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.413664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.413622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:08.421074 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.421039 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:08:08.689627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:08.689600 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:08:08.691399 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:08:08.691373 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcadfd41c_4397_4982_a4e8_7346fac046e6.slice/crio-7aa86b84b22bd3143f3cee138b4a5d5200ca9ef5ed9c8f3cd0baf80aec4188ac WatchSource:0}: Error finding container 7aa86b84b22bd3143f3cee138b4a5d5200ca9ef5ed9c8f3cd0baf80aec4188ac: Status 404 returned error can't find the container with id 7aa86b84b22bd3143f3cee138b4a5d5200ca9ef5ed9c8f3cd0baf80aec4188ac Sep 20 02:08:09.005674 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:09.005576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" event={"ID":"cadfd41c-4397-4982-a4e8-7346fac046e6","Type":"ContainerStarted","Data":"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a"} Sep 20 02:08:09.005674 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:09.005626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" event={"ID":"cadfd41c-4397-4982-a4e8-7346fac046e6","Type":"ContainerStarted","Data":"7aa86b84b22bd3143f3cee138b4a5d5200ca9ef5ed9c8f3cd0baf80aec4188ac"} Sep 20 02:08:09.023838 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:09.023781 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" podStartSLOduration=1.023766862 podStartE2EDuration="1.023766862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:08:09.023281417 +0000 UTC m=+940.541597174" watchObservedRunningTime="2026-09-20 02:08:09.023766862 +0000 UTC m=+940.542082621" Sep 20 02:08:18.414195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:18.414155 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:18.414568 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:18.414221 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:18.426927 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:18.426885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:19.074987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.074956 2578 generic.go:356] "Generic (PLEG): container finished" podID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerID="41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c" exitCode=0 Sep 20 02:08:19.075160 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.075028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerDied","Data":"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c"} Sep 20 02:08:19.075491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.075457 2578 scope.go:117] "RemoveContainer" containerID="41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c" Sep 20 02:08:19.076765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.076739 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:19.076839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.076802 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:19.076878 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.076837 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:19.089973 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.089954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:08:19.707359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.707320 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:08:19.708002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.707979 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:19.708002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.708003 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e65ac6f-0bec-4d0c-82ca-4e1914641704" containerName="istio-proxy" Sep 20 02:08:19.710324 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.710303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.713509 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.713488 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4cfa5a\"/\"kube-root-ca.crt\"" Sep 20 02:08:19.713630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.713491 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4cfa5a\"/\"default-dockercfg-69d4d\"" Sep 20 02:08:19.713630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.713492 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4cfa5a\"/\"autores-model-4cfa5a-kserve-self-signed-certs\"" Sep 20 02:08:19.713630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.713491 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4cfa5a\"/\"openshift-service-ca.crt\"" Sep 20 02:08:19.719615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.719593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:08:19.791440 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.791612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.791612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hft\" (UniqueName: \"kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.791747 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.791747 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.791858 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.791754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hft\" (UniqueName: \"kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.892995 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.892972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.893227 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.893166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.893308 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.893281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.893476 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.893458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.895317 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.895298 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.895464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.895449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:19.900103 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:19.900081 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hft\" (UniqueName: \"kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft\") pod \"autores-model-4cfa5a-kserve-db7776fc9-h8xnl\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:20.021123 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.021043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:20.093011 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.092371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerStarted","Data":"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813"} Sep 20 02:08:20.299892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.299868 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:08:20.301187 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:08:20.301159 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da78b86_dc48_4a66_8c29_df0ac7e79239.slice/crio-9d70d5093a079de88f1675491b9998b2afc1ac3464a2888a57fa5c3e56541a07 WatchSource:0}: Error finding container 9d70d5093a079de88f1675491b9998b2afc1ac3464a2888a57fa5c3e56541a07: Status 404 returned error can't find the container with id 9d70d5093a079de88f1675491b9998b2afc1ac3464a2888a57fa5c3e56541a07 Sep 20 02:08:20.748463 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.748421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:20.749033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.748553 2578 patch_prober.go:28] interesting pod/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 20 02:08:20.749033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:20.748598 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 20 02:08:21.099101 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:21.099011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" event={"ID":"5da78b86-dc48-4a66-8c29-df0ac7e79239","Type":"ContainerStarted","Data":"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91"} Sep 20 02:08:21.099101 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:21.099055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" event={"ID":"5da78b86-dc48-4a66-8c29-df0ac7e79239","Type":"ContainerStarted","Data":"9d70d5093a079de88f1675491b9998b2afc1ac3464a2888a57fa5c3e56541a07"} Sep 20 02:08:21.114655 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:21.114609 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" podStartSLOduration=2.114595692 podStartE2EDuration="2.114595692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:08:21.113303766 +0000 UTC m=+952.631619525" watchObservedRunningTime="2026-09-20 02:08:21.114595692 +0000 UTC m=+952.632911521" Sep 20 02:08:21.748952 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:21.748884 2578 patch_prober.go:28] interesting pod/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 20 02:08:21.749417 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:21.748977 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 20 02:08:22.748287 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:22.748242 2578 patch_prober.go:28] interesting pod/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 20 02:08:22.748503 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:22.748314 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 20 02:08:23.748837 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:23.748791 2578 patch_prober.go:28] interesting pod/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 20 02:08:23.749357 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:23.748864 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 20 02:08:24.748851 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:24.748804 2578 patch_prober.go:28] interesting pod/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 20 02:08:24.749371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:24.748878 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 20 02:08:25.965767 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:25.965723 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:25.966334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:25.966309 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:25.967401 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:25.967374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:26.615410 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:26.615376 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:08:29.112988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.112960 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:08:29.113414 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.113164 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" podUID="7cef3897-0f09-4b7a-830a-227ccd405138" containerName="maas-api" containerID="cri-o://5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24" gracePeriod=30 Sep 20 02:08:29.368241 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.368217 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:29.449651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.449614 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:08:29.449920 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.449858 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" containerID="cri-o://a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813" gracePeriod=30 Sep 20 02:08:29.493533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.493488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca\") pod \"7cef3897-0f09-4b7a-830a-227ccd405138\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " Sep 20 02:08:29.493709 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.493589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kl6w\" (UniqueName: \"kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w\") pod \"7cef3897-0f09-4b7a-830a-227ccd405138\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " Sep 20 02:08:29.493709 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.493652 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") pod \"7cef3897-0f09-4b7a-830a-227ccd405138\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " Sep 20 02:08:29.493709 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.493682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs\") pod \"7cef3897-0f09-4b7a-830a-227ccd405138\" (UID: \"7cef3897-0f09-4b7a-830a-227ccd405138\") " Sep 20 02:08:29.493876 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.493846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cef3897-0f09-4b7a-830a-227ccd405138" (UID: "7cef3897-0f09-4b7a-830a-227ccd405138"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:08:29.494146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.494118 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef3897-0f09-4b7a-830a-227ccd405138-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:29.495880 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.495848 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cef3897-0f09-4b7a-830a-227ccd405138" (UID: "7cef3897-0f09-4b7a-830a-227ccd405138"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:08:29.496076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.495994 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7cef3897-0f09-4b7a-830a-227ccd405138" (UID: "7cef3897-0f09-4b7a-830a-227ccd405138"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:08:29.496076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.496010 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w" (OuterVolumeSpecName: "kube-api-access-4kl6w") pod "7cef3897-0f09-4b7a-830a-227ccd405138" (UID: "7cef3897-0f09-4b7a-830a-227ccd405138"). InnerVolumeSpecName "kube-api-access-4kl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:08:29.594815 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.594778 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kl6w\" (UniqueName: \"kubernetes.io/projected/7cef3897-0f09-4b7a-830a-227ccd405138-kube-api-access-4kl6w\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:29.594815 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.594809 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:29.594815 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:29.594818 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cef3897-0f09-4b7a-830a-227ccd405138-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:30.021923 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.021885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:30.022129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.021946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:30.034167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.034144 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:30.158177 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.158143 2578 generic.go:356] "Generic (PLEG): container finished" podID="7cef3897-0f09-4b7a-830a-227ccd405138" containerID="5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24" exitCode=0 Sep 20 02:08:30.158607 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.158190 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" Sep 20 02:08:30.158607 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.158226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" event={"ID":"7cef3897-0f09-4b7a-830a-227ccd405138","Type":"ContainerDied","Data":"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24"} Sep 20 02:08:30.158607 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.158285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn" event={"ID":"7cef3897-0f09-4b7a-830a-227ccd405138","Type":"ContainerDied","Data":"f01bba01e2555c5bf83e3949911b6cdb336365fa2e4d3bc8f2dd1853642588c7"} Sep 20 02:08:30.158607 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.158315 2578 scope.go:117] "RemoveContainer" containerID="5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24" Sep 20 02:08:30.169595 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.169576 2578 scope.go:117] "RemoveContainer" containerID="5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24" Sep 20 02:08:30.170117 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:08:30.170057 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24\": container with ID starting with 5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24 not found: ID does not exist" containerID="5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24" Sep 20 02:08:30.170117 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.170091 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24"} err="failed to get container status \"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24\": rpc error: code = NotFound desc = could not find container \"5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24\": container with ID starting with 5c2cc2b7415f36001082616f8eaadf55ea3af22b3032afeb39f6e42a89c64c24 not found: ID does not exist" Sep 20 02:08:30.173102 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.173082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:08:30.187829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.187805 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:08:30.193189 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.193167 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-34c720e3-7889f4fb6b-qp4pn"] Sep 20 02:08:30.253897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.253867 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:08:30.254472 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.254447 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cef3897-0f09-4b7a-830a-227ccd405138" containerName="maas-api" Sep 20 02:08:30.254613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.254479 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cef3897-0f09-4b7a-830a-227ccd405138" containerName="maas-api" Sep 20 02:08:30.254613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.254601 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cef3897-0f09-4b7a-830a-227ccd405138" containerName="maas-api" Sep 20 02:08:30.257508 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.257489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.259698 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.259669 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4c51cf40-openshift-default-dockercfg-gsqbj\"" Sep 20 02:08:30.281412 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.281321 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:08:30.401826 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.401788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.401840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.401952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402122 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lcx\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402193 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402227 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.402271 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.402256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503553 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lcx\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.503757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.503841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504219 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.504139 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504315 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.504290 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504467 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.504447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504513 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.504445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.504656 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.504636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.506205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.506181 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.506663 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.506647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.512691 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.512667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lcx\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.512836 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.512815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token\") pod \"e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.569388 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.569300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:30.857710 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.857680 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:08:30.858933 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:08:30.858878 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f50be9_a1bc_4613_a9ea_503e2ca5474c.slice/crio-d40edcd0f707d6c4c4735814fbed006f976865d9d6b605d6e538eba200e2cf65 WatchSource:0}: Error finding container d40edcd0f707d6c4c4735814fbed006f976865d9d6b605d6e538eba200e2cf65: Status 404 returned error can't find the container with id d40edcd0f707d6c4c4735814fbed006f976865d9d6b605d6e538eba200e2cf65 Sep 20 02:08:30.861499 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.861469 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:30.861572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.861531 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:30.861572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:30.861559 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:31.017816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.017780 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cef3897-0f09-4b7a-830a-227ccd405138" path="/var/lib/kubelet/pods/7cef3897-0f09-4b7a-830a-227ccd405138/volumes" Sep 20 02:08:31.166730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.166690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerStarted","Data":"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b"} Sep 20 02:08:31.167203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.166759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerStarted","Data":"d40edcd0f707d6c4c4735814fbed006f976865d9d6b605d6e538eba200e2cf65"} Sep 20 02:08:31.192617 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.192573 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podStartSLOduration=1.192557573 podStartE2EDuration="1.192557573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:08:31.191208083 +0000 UTC m=+962.709523842" watchObservedRunningTime="2026-09-20 02:08:31.192557573 +0000 UTC m=+962.710873332" Sep 20 02:08:31.570231 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.570151 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:31.575505 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:31.575481 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:32.173997 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:32.173963 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:32.174998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:32.174982 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:08:34.701090 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.701065 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:34.844745 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844651 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.844745 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844712 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844773 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844888 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.844993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjd7s\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s\") pod \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\" (UID: \"8e5cc6b3-327b-48bf-89dd-dc40852fb92f\") " Sep 20 02:08:34.845394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845090 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:08:34.845394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845099 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:08:34.845394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845116 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:08:34.845394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845177 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:08:34.845394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845273 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:08:34.845649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845462 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.845649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845483 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.845649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845497 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.845649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845510 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.845649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.845522 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.847163 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.847137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:08:34.847265 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.847176 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s" (OuterVolumeSpecName: "kube-api-access-vjd7s") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "kube-api-access-vjd7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:08:34.847630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.847611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:08:34.847674 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.847615 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e5cc6b3-327b-48bf-89dd-dc40852fb92f" (UID: "8e5cc6b3-327b-48bf-89dd-dc40852fb92f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:08:34.946984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.946956 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.946984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.946978 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.946984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.946987 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:34.947175 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:34.946996 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjd7s\" (UniqueName: \"kubernetes.io/projected/8e5cc6b3-327b-48bf-89dd-dc40852fb92f-kube-api-access-vjd7s\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:08:35.021189 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021157 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:08:35.021746 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021728 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:08:35.021845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021748 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:08:35.021845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021759 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:08:35.021845 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021767 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:08:35.022055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.021878 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:08:35.024550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.024529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.026589 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.026566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4c51cf40-serving-cert\"" Sep 20 02:08:35.035404 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.035379 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:08:35.148478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.148442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.148651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.148499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd77f\" (UniqueName: \"kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.148651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.148563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.148726 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.148660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.194161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.194129 2578 generic.go:356] "Generic (PLEG): container finished" podID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerID="a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813" exitCode=0 Sep 20 02:08:35.194323 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.194193 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" Sep 20 02:08:35.194323 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.194214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerDied","Data":"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813"} Sep 20 02:08:35.194323 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.194270 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm" event={"ID":"8e5cc6b3-327b-48bf-89dd-dc40852fb92f","Type":"ContainerDied","Data":"d47916bf1c326ad680224df9d10fe3810a77789abc3b2cc7ff3bdf6a55a60948"} Sep 20 02:08:35.194323 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.194291 2578 scope.go:117] "RemoveContainer" containerID="a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813" Sep 20 02:08:35.205053 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.205030 2578 scope.go:117] "RemoveContainer" containerID="41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c" Sep 20 02:08:35.211753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.211729 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:08:35.214613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.214587 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-34c720e3-gw-openshift-default-7976458dbd-5ztqm"] Sep 20 02:08:35.215087 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.215069 2578 scope.go:117] "RemoveContainer" containerID="a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813" Sep 20 02:08:35.215377 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:08:35.215357 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813\": container with ID starting with a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813 not found: ID does not exist" containerID="a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813" Sep 20 02:08:35.215422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.215389 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813"} err="failed to get container status \"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813\": rpc error: code = NotFound desc = could not find container \"a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813\": container with ID starting with a34a796a9f5444d371211665a12fc28b7dac1250aca463cc9bc3b590255af813 not found: ID does not exist" Sep 20 02:08:35.215422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.215407 2578 scope.go:117] "RemoveContainer" containerID="41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c" Sep 20 02:08:35.215684 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:08:35.215667 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c\": container with ID starting with 41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c not found: ID does not exist" containerID="41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c" Sep 20 02:08:35.215745 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.215689 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c"} err="failed to get container status \"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c\": rpc error: code = NotFound desc = could not find container \"41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c\": container with ID starting with 41383d3a77c9a714dd05ad6e57325492b13efd456c85da9e0d9c3e876817fa3c not found: ID does not exist" Sep 20 02:08:35.250208 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.250181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.250311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.250238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.250311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.250293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.250420 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.250321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd77f\" (UniqueName: \"kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.250474 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:08:35.250428 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-serving-cert: secret "maas-api-e2e-ait-4c51cf40-serving-cert" not found Sep 20 02:08:35.250522 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:08:35.250503 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls podName:dfa380c6-53d6-452d-84eb-d4d1b9f5205f nodeName:}" failed. No retries permitted until 2026-09-20 02:08:35.750481645 +0000 UTC m=+967.268797394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls") pod "maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" (UID: "dfa380c6-53d6-452d-84eb-d4d1b9f5205f") : secret "maas-api-e2e-ait-4c51cf40-serving-cert" not found Sep 20 02:08:35.250923 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.250885 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.252788 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.252768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.258767 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.258746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd77f\" (UniqueName: \"kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.754539 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.754505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.757100 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.757077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") pod \"maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:35.950660 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:35.950624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:36.221461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:36.221435 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:08:36.224221 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:08:36.224195 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa380c6_53d6_452d_84eb_d4d1b9f5205f.slice/crio-ef88f83f4f957bf03234e7fb4e9b4193eaf090afa8d7390f260b5f9c707e1fd7 WatchSource:0}: Error finding container ef88f83f4f957bf03234e7fb4e9b4193eaf090afa8d7390f260b5f9c707e1fd7: Status 404 returned error can't find the container with id ef88f83f4f957bf03234e7fb4e9b4193eaf090afa8d7390f260b5f9c707e1fd7 Sep 20 02:08:37.016374 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:37.016335 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" path="/var/lib/kubelet/pods/8e5cc6b3-327b-48bf-89dd-dc40852fb92f/volumes" Sep 20 02:08:37.209063 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:37.209024 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" event={"ID":"dfa380c6-53d6-452d-84eb-d4d1b9f5205f","Type":"ContainerStarted","Data":"dd4a9696bf351b343c8d930c38cb8e5fec1d75620325d95409826c28800bb319"} Sep 20 02:08:37.209252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:37.209074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" event={"ID":"dfa380c6-53d6-452d-84eb-d4d1b9f5205f","Type":"ContainerStarted","Data":"ef88f83f4f957bf03234e7fb4e9b4193eaf090afa8d7390f260b5f9c707e1fd7"} Sep 20 02:08:37.209252 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:37.209113 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:37.226098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:37.226052 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" podStartSLOduration=2.8632320140000003 podStartE2EDuration="3.226037033s" totalImagesPullingTime="362.805019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:08:37.22492528 +0000 UTC m=+968.743241029" watchObservedRunningTime="2026-09-20 02:08:37.226037033 +0000 UTC m=+968.744352847" Sep 20 02:08:43.221372 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:43.221341 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:08:49.318412 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:49.318376 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:08:59.350105 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.350072 2578 generic.go:356] "Generic (PLEG): container finished" podID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerID="775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b" exitCode=0 Sep 20 02:08:59.350496 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.350112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerDied","Data":"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b"} Sep 20 02:08:59.350541 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.350518 2578 scope.go:117] "RemoveContainer" containerID="775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b" Sep 20 02:08:59.351763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.351723 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:59.351864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.351807 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:08:59.351864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:08:59.351853 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:00.362476 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:00.362442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerStarted","Data":"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e"} Sep 20 02:09:00.569739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:00.569696 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:00.570898 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:00.570873 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:00.571031 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:00.570938 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:01.570280 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:01.570241 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:01.570699 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:01.570301 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:02.174546 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:02.174508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:02.570137 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:02.570036 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:02.570286 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:02.570124 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:03.569847 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:03.569812 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:03.570242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:03.569865 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:04.570401 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:04.570360 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:04.570794 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:04.570424 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:05.570629 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:05.570588 2578 patch_prober.go:28] interesting pod/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 20 02:09:05.571047 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:05.570658 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 20 02:09:05.975172 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:05.975136 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:09:05.975406 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:05.975379 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" podUID="f2648736-4025-4bd2-b349-7c268d5efa54" containerName="maas-api" containerID="cri-o://723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd" gracePeriod=30 Sep 20 02:09:06.237747 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.237679 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:09:06.363733 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.363702 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") pod \"f2648736-4025-4bd2-b349-7c268d5efa54\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " Sep 20 02:09:06.363733 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.363739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca\") pod \"f2648736-4025-4bd2-b349-7c268d5efa54\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " Sep 20 02:09:06.363975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.363825 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs\") pod \"f2648736-4025-4bd2-b349-7c268d5efa54\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " Sep 20 02:09:06.363975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.363856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l\") pod \"f2648736-4025-4bd2-b349-7c268d5efa54\" (UID: \"f2648736-4025-4bd2-b349-7c268d5efa54\") " Sep 20 02:09:06.364183 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.364153 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2648736-4025-4bd2-b349-7c268d5efa54" (UID: "f2648736-4025-4bd2-b349-7c268d5efa54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:09:06.366235 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.366201 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2648736-4025-4bd2-b349-7c268d5efa54" (UID: "f2648736-4025-4bd2-b349-7c268d5efa54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:09:06.366359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.366236 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l" (OuterVolumeSpecName: "kube-api-access-hrx9l") pod "f2648736-4025-4bd2-b349-7c268d5efa54" (UID: "f2648736-4025-4bd2-b349-7c268d5efa54"). InnerVolumeSpecName "kube-api-access-hrx9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:06.366359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.366311 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2648736-4025-4bd2-b349-7c268d5efa54" (UID: "f2648736-4025-4bd2-b349-7c268d5efa54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:09:06.404719 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.404676 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2648736-4025-4bd2-b349-7c268d5efa54" containerID="723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd" exitCode=0 Sep 20 02:09:06.404922 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.404733 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" Sep 20 02:09:06.404922 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.404776 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" event={"ID":"f2648736-4025-4bd2-b349-7c268d5efa54","Type":"ContainerDied","Data":"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd"} Sep 20 02:09:06.405033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.404956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd" event={"ID":"f2648736-4025-4bd2-b349-7c268d5efa54","Type":"ContainerDied","Data":"57e116ab1d7622bfa22d7d1fb16522877baef8d8ff0064681d5b1cd8a332d077"} Sep 20 02:09:06.405033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.405020 2578 scope.go:117] "RemoveContainer" containerID="723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd" Sep 20 02:09:06.422272 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.422250 2578 scope.go:117] "RemoveContainer" containerID="723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd" Sep 20 02:09:06.422718 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:06.422684 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd\": container with ID starting with 723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd not found: ID does not exist" containerID="723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd" Sep 20 02:09:06.422829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.422718 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd"} err="failed to get container status \"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd\": rpc error: code = NotFound desc = could not find container \"723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd\": container with ID starting with 723ef61a01229fc18b6342806d45ebc5364234cf611c43a4d9147ef58458c6bd not found: ID does not exist" Sep 20 02:09:06.432021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.431988 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:09:06.435485 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.435452 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4cfa5a-688fc678bf-9gvgd"] Sep 20 02:09:06.465085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.465049 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:06.465085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.465077 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrx9l\" (UniqueName: \"kubernetes.io/projected/f2648736-4025-4bd2-b349-7c268d5efa54-kube-api-access-hrx9l\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:06.465085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.465087 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2648736-4025-4bd2-b349-7c268d5efa54-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:06.465326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.465097 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2648736-4025-4bd2-b349-7c268d5efa54-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:06.574829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.574747 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:06.575945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:06.575901 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:07.017426 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:07.017389 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2648736-4025-4bd2-b349-7c268d5efa54" path="/var/lib/kubelet/pods/f2648736-4025-4bd2-b349-7c268d5efa54/volumes" Sep 20 02:09:11.555449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:11.555409 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:09:14.241231 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.241191 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:09:14.241636 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.241445 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" podUID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" containerName="maas-api" containerID="cri-o://dd4a9696bf351b343c8d930c38cb8e5fec1d75620325d95409826c28800bb319" gracePeriod=30 Sep 20 02:09:14.455408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.455374 2578 generic.go:356] "Generic (PLEG): container finished" podID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" containerID="dd4a9696bf351b343c8d930c38cb8e5fec1d75620325d95409826c28800bb319" exitCode=0 Sep 20 02:09:14.455591 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.455405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" event={"ID":"dfa380c6-53d6-452d-84eb-d4d1b9f5205f","Type":"ContainerDied","Data":"dd4a9696bf351b343c8d930c38cb8e5fec1d75620325d95409826c28800bb319"} Sep 20 02:09:14.491387 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.491326 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:09:14.650847 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.650807 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd77f\" (UniqueName: \"kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f\") pod \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " Sep 20 02:09:14.651078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.650856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") pod \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " Sep 20 02:09:14.651078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.650934 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca\") pod \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " Sep 20 02:09:14.651078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.651037 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs\") pod \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\" (UID: \"dfa380c6-53d6-452d-84eb-d4d1b9f5205f\") " Sep 20 02:09:14.651362 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.651333 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfa380c6-53d6-452d-84eb-d4d1b9f5205f" (UID: "dfa380c6-53d6-452d-84eb-d4d1b9f5205f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:09:14.653398 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.653361 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfa380c6-53d6-452d-84eb-d4d1b9f5205f" (UID: "dfa380c6-53d6-452d-84eb-d4d1b9f5205f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:09:14.653398 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.653383 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfa380c6-53d6-452d-84eb-d4d1b9f5205f" (UID: "dfa380c6-53d6-452d-84eb-d4d1b9f5205f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:09:14.653558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.653467 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f" (OuterVolumeSpecName: "kube-api-access-kd77f") pod "dfa380c6-53d6-452d-84eb-d4d1b9f5205f" (UID: "dfa380c6-53d6-452d-84eb-d4d1b9f5205f"). InnerVolumeSpecName "kube-api-access-kd77f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:14.721467 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.721417 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:09:14.752531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.752443 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:14.752531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.752478 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd77f\" (UniqueName: \"kubernetes.io/projected/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-kube-api-access-kd77f\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:14.752531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.752494 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:14.752531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:14.752504 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa380c6-53d6-452d-84eb-d4d1b9f5205f-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:15.463460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.463431 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" Sep 20 02:09:15.463957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.463460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr" event={"ID":"dfa380c6-53d6-452d-84eb-d4d1b9f5205f","Type":"ContainerDied","Data":"ef88f83f4f957bf03234e7fb4e9b4193eaf090afa8d7390f260b5f9c707e1fd7"} Sep 20 02:09:15.463957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.463528 2578 scope.go:117] "RemoveContainer" containerID="dd4a9696bf351b343c8d930c38cb8e5fec1d75620325d95409826c28800bb319" Sep 20 02:09:15.498202 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.498168 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:09:15.502549 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.502522 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c51cf40-6759dc7884-2ftqr"] Sep 20 02:09:15.626872 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.626839 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:09:15.627138 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:15.627096 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" containerID="cri-o://b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e" gracePeriod=30 Sep 20 02:09:16.341024 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.340992 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:09:16.341574 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341559 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" containerName="maas-api" Sep 20 02:09:16.341631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341579 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" containerName="maas-api" Sep 20 02:09:16.341631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341619 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2648736-4025-4bd2-b349-7c268d5efa54" containerName="maas-api" Sep 20 02:09:16.341631 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341627 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2648736-4025-4bd2-b349-7c268d5efa54" containerName="maas-api" Sep 20 02:09:16.341735 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341724 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2648736-4025-4bd2-b349-7c268d5efa54" containerName="maas-api" Sep 20 02:09:16.341771 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341734 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" containerName="maas-api" Sep 20 02:09:16.341771 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.341749 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5cc6b3-327b-48bf-89dd-dc40852fb92f" containerName="istio-proxy" Sep 20 02:09:16.344853 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.344831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.347192 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.347169 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-066f62f9-openshift-default-dockercfg-twl69\"" Sep 20 02:09:16.360036 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.360004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:09:16.470587 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.470970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.471021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6hr\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.471116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.471047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.571980 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.571933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572369 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572369 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572369 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572369 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6hr\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.572962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.572942 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.574772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.574739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.574865 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.574840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.580208 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.580178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.580301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.580289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6hr\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr\") pod \"e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.657288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.657248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:16.935900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.935855 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:09:16.937156 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:09:16.937125 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f26963c_604d_4180_937a_efafdb7a2cee.slice/crio-02df7186774a1eaa53de6808abb2b798b0d82e1b3e74c835d795849ad30b2877 WatchSource:0}: Error finding container 02df7186774a1eaa53de6808abb2b798b0d82e1b3e74c835d795849ad30b2877: Status 404 returned error can't find the container with id 02df7186774a1eaa53de6808abb2b798b0d82e1b3e74c835d795849ad30b2877 Sep 20 02:09:16.939664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.939634 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:16.939737 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.939699 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:16.939737 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:16.939730 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:17.016992 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.016963 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa380c6-53d6-452d-84eb-d4d1b9f5205f" path="/var/lib/kubelet/pods/dfa380c6-53d6-452d-84eb-d4d1b9f5205f/volumes" Sep 20 02:09:17.479936 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.479867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerStarted","Data":"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7"} Sep 20 02:09:17.480413 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.479960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerStarted","Data":"02df7186774a1eaa53de6808abb2b798b0d82e1b3e74c835d795849ad30b2877"} Sep 20 02:09:17.503142 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.503085 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podStartSLOduration=1.503065936 podStartE2EDuration="1.503065936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:09:17.500202629 +0000 UTC m=+1009.018518385" watchObservedRunningTime="2026-09-20 02:09:17.503065936 +0000 UTC m=+1009.021381693" Sep 20 02:09:17.657620 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.657585 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:17.663200 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:17.663170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:18.486660 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:18.486629 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:18.487671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:18.487654 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:20.884385 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:20.884358 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:21.025110 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025024 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025110 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025110 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025110 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2lcx\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025239 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025326 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025352 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket\") pod \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\" (UID: \"17f50be9-a1bc-4613-a9ea-503e2ca5474c\") " Sep 20 02:09:21.025726 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025487 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:21.025726 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025570 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data" (OuterVolumeSpecName: "istio-data") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:21.025726 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025712 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:21.025919 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025762 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:09:21.026011 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025988 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:21.026085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.025956 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.026085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.026050 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.026085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.026063 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.026085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.026076 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.027575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.027553 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token" (OuterVolumeSpecName: "istio-token") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:21.027679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.027569 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:21.027679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.027600 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx" (OuterVolumeSpecName: "kube-api-access-t2lcx") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "kube-api-access-t2lcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:21.027834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.027807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17f50be9-a1bc-4613-a9ea-503e2ca5474c" (UID: "17f50be9-a1bc-4613-a9ea-503e2ca5474c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:09:21.127494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.127459 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.127494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.127488 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.127494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.127496 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.127494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.127504 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2lcx\" (UniqueName: \"kubernetes.io/projected/17f50be9-a1bc-4613-a9ea-503e2ca5474c-kube-api-access-t2lcx\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.127800 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.127525 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f50be9-a1bc-4613-a9ea-503e2ca5474c-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:21.513460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.513417 2578 generic.go:356] "Generic (PLEG): container finished" podID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerID="b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e" exitCode=0 Sep 20 02:09:21.513667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.513492 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" Sep 20 02:09:21.513667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.513507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerDied","Data":"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e"} Sep 20 02:09:21.513667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.513606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt" event={"ID":"17f50be9-a1bc-4613-a9ea-503e2ca5474c","Type":"ContainerDied","Data":"d40edcd0f707d6c4c4735814fbed006f976865d9d6b605d6e538eba200e2cf65"} Sep 20 02:09:21.513667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.513636 2578 scope.go:117] "RemoveContainer" containerID="b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e" Sep 20 02:09:21.525857 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.525830 2578 scope.go:117] "RemoveContainer" containerID="775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b" Sep 20 02:09:21.537480 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.537446 2578 scope.go:117] "RemoveContainer" containerID="b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e" Sep 20 02:09:21.537835 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:21.537811 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e\": container with ID starting with b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e not found: ID does not exist" containerID="b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e" Sep 20 02:09:21.537945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.537850 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e"} err="failed to get container status \"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e\": rpc error: code = NotFound desc = could not find container \"b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e\": container with ID starting with b85f7163f3c1d3b5f117e35b31e09e4ecc7201e019793e1fd1e7417efc70827e not found: ID does not exist" Sep 20 02:09:21.537945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.537877 2578 scope.go:117] "RemoveContainer" containerID="775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b" Sep 20 02:09:21.538218 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:21.538196 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b\": container with ID starting with 775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b not found: ID does not exist" containerID="775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b" Sep 20 02:09:21.538266 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.538228 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b"} err="failed to get container status \"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b\": rpc error: code = NotFound desc = could not find container \"775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b\": container with ID starting with 775c1269e043ea6e1e06cf3e678994b7f26ebfd08d415ac6e159319721ab096b not found: ID does not exist" Sep 20 02:09:21.538811 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.538787 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:09:21.541871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:21.541849 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4c51cf40-openshift-default-78c989476d-9kzbt"] Sep 20 02:09:22.115559 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.115524 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:09:22.116082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.115951 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:22.116082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.115964 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:22.116082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.116069 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:22.116082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.116078 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:22.119074 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.119053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.121245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.121221 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 20 02:09:22.122310 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.122150 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 20 02:09:22.122310 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.122177 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 20 02:09:22.122310 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.122187 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z4spz\"" Sep 20 02:09:22.122544 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.122451 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-066f62f9-serving-cert\"" Sep 20 02:09:22.130794 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.130737 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:09:22.239654 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.239616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.239654 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.239662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84q7\" (UniqueName: \"kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.240003 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.239850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.240003 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.239886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.341178 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.341138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.341178 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.341182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84q7\" (UniqueName: \"kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.341414 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.341265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.341414 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.341291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.341971 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.341951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.343703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.343685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.343849 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.343830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.349131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.349107 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84q7\" (UniqueName: \"kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7\") pod \"maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.434035 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.433980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:22.716621 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.716593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:09:22.719149 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:09:22.719123 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53351b46_7d9a_4523_b649_aec05a0ae5c5.slice/crio-b105ba03ed738284e2893249fdee7871728faba96e9ec194175f078ad557112f WatchSource:0}: Error finding container b105ba03ed738284e2893249fdee7871728faba96e9ec194175f078ad557112f: Status 404 returned error can't find the container with id b105ba03ed738284e2893249fdee7871728faba96e9ec194175f078ad557112f Sep 20 02:09:22.721157 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:22.721136 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 02:09:23.016046 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:23.015959 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" path="/var/lib/kubelet/pods/17f50be9-a1bc-4613-a9ea-503e2ca5474c/volumes" Sep 20 02:09:23.530072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:23.530039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" event={"ID":"53351b46-7d9a-4523-b649-aec05a0ae5c5","Type":"ContainerStarted","Data":"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe"} Sep 20 02:09:23.530466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:23.530092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" event={"ID":"53351b46-7d9a-4523-b649-aec05a0ae5c5","Type":"ContainerStarted","Data":"b105ba03ed738284e2893249fdee7871728faba96e9ec194175f078ad557112f"} Sep 20 02:09:23.530466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:23.530114 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:23.547092 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:23.547037 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" podStartSLOduration=1.1402112 podStartE2EDuration="1.54701749s" totalImagesPullingTime="406.80629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:09:23.544596713 +0000 UTC m=+1015.062912486" watchObservedRunningTime="2026-09-20 02:09:23.54701749 +0000 UTC m=+1015.065333248" Sep 20 02:09:26.576869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.576782 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" podUID="5da78b86-dc48-4a66-8c29-df0ac7e79239" containerName="main" containerID="cri-o://ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91" gracePeriod=45 Sep 20 02:09:26.831108 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.831034 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:09:26.881358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881319 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881561 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881370 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881561 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881424 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881561 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881446 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881561 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881526 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881577 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4hft\" (UniqueName: \"kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft\") pod \"5da78b86-dc48-4a66-8c29-df0ac7e79239\" (UID: \"5da78b86-dc48-4a66-8c29-df0ac7e79239\") " Sep 20 02:09:26.881772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881660 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache" (OuterVolumeSpecName: "model-cache") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:26.881772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881652 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home" (OuterVolumeSpecName: "home") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:26.881944 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.881835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:26.882019 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.882001 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:26.882055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.882025 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:26.882055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.882039 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:26.883863 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.883835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:09:26.884275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.884252 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft" (OuterVolumeSpecName: "kube-api-access-h4hft") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "kube-api-access-h4hft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:26.884341 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.884250 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm" (OuterVolumeSpecName: "dshm") pod "5da78b86-dc48-4a66-8c29-df0ac7e79239" (UID: "5da78b86-dc48-4a66-8c29-df0ac7e79239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:26.983506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.983457 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hft\" (UniqueName: \"kubernetes.io/projected/5da78b86-dc48-4a66-8c29-df0ac7e79239-kube-api-access-h4hft\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:26.983506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.983502 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da78b86-dc48-4a66-8c29-df0ac7e79239-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:26.983506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:26.983517 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da78b86-dc48-4a66-8c29-df0ac7e79239-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:27.556207 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.556169 2578 generic.go:356] "Generic (PLEG): container finished" podID="5da78b86-dc48-4a66-8c29-df0ac7e79239" containerID="ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91" exitCode=0 Sep 20 02:09:27.556207 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.556210 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" Sep 20 02:09:27.556455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.556239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" event={"ID":"5da78b86-dc48-4a66-8c29-df0ac7e79239","Type":"ContainerDied","Data":"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91"} Sep 20 02:09:27.556455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.556281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl" event={"ID":"5da78b86-dc48-4a66-8c29-df0ac7e79239","Type":"ContainerDied","Data":"9d70d5093a079de88f1675491b9998b2afc1ac3464a2888a57fa5c3e56541a07"} Sep 20 02:09:27.556455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.556297 2578 scope.go:117] "RemoveContainer" containerID="ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91" Sep 20 02:09:27.566149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.566131 2578 scope.go:117] "RemoveContainer" containerID="ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91" Sep 20 02:09:27.566413 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:27.566398 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91\": container with ID starting with ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91 not found: ID does not exist" containerID="ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91" Sep 20 02:09:27.566473 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.566421 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91"} err="failed to get container status \"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91\": rpc error: code = NotFound desc = could not find container \"ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91\": container with ID starting with ca8280c2e22b2cb7f43988cbae07e48c72985668ce7af16abb12fe41a940ab91 not found: ID does not exist" Sep 20 02:09:27.573017 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.572994 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:09:27.577152 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:27.577127 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4cfa5a/autores-model-4cfa5a-kserve-db7776fc9-h8xnl"] Sep 20 02:09:29.017415 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:29.017382 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da78b86-dc48-4a66-8c29-df0ac7e79239" path="/var/lib/kubelet/pods/5da78b86-dc48-4a66-8c29-df0ac7e79239/volumes" Sep 20 02:09:29.542296 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:29.542264 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:09:32.827885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:32.827835 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:09:43.664486 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.664451 2578 generic.go:356] "Generic (PLEG): container finished" podID="5f26963c-604d-4180-937a-efafdb7a2cee" containerID="20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7" exitCode=0 Sep 20 02:09:43.664988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.664505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerDied","Data":"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7"} Sep 20 02:09:43.664988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.664944 2578 scope.go:117] "RemoveContainer" containerID="20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7" Sep 20 02:09:43.666054 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.666025 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:43.666165 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.666091 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:43.666165 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:43.666127 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:44.673206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:44.673166 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerStarted","Data":"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9"} Sep 20 02:09:45.657729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:45.657681 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:45.659223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:45.659192 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:45.659357 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:45.659258 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:46.658533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:46.658494 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:46.659174 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:46.659133 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:47.658368 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:47.658317 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:47.658595 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:47.658396 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:48.487667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:48.487631 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:48.658075 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:48.658030 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:48.658267 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:48.658102 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:48.698312 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:48.698282 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:09:48.698752 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:48.698504 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" containerID="cri-o://9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7" gracePeriod=30 Sep 20 02:09:49.480347 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.480311 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:09:49.481199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.481180 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:49.481315 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.481304 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f50be9-a1bc-4613-a9ea-503e2ca5474c" containerName="istio-proxy" Sep 20 02:09:49.481424 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.481414 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da78b86-dc48-4a66-8c29-df0ac7e79239" containerName="main" Sep 20 02:09:49.481502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.481494 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da78b86-dc48-4a66-8c29-df0ac7e79239" containerName="main" Sep 20 02:09:49.481673 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.481663 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da78b86-dc48-4a66-8c29-df0ac7e79239" containerName="main" Sep 20 02:09:49.484718 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.484693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.487235 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.487215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-725649-openshift-default-dockercfg-w6qql\"" Sep 20 02:09:49.495690 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.495663 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:09:49.615671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.615951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.615951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.615951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.615951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.616197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.615960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xpmk\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.616197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.616015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.616197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.616052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.616197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.616082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.658158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.658117 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:49.658458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.658430 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:49.717381 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717381 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xpmk\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.717801 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.717541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.718396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.718237 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.718396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.718357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.718396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.718371 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.718872 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.718845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.719180 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.718851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.720108 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.720040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.720897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.720879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.725979 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.725960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xpmk\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.726084 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.726054 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token\") pod \"e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:49.797436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:49.797338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:50.098625 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.098592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:09:50.099564 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:09:50.099536 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeee7480_4894_4886_a9fd_1f8c76bf3774.slice/crio-e1c2ca7c8279ebf2d05f072e1ab7d747193db6daa432955a28dd3505a85d0a59 WatchSource:0}: Error finding container e1c2ca7c8279ebf2d05f072e1ab7d747193db6daa432955a28dd3505a85d0a59: Status 404 returned error can't find the container with id e1c2ca7c8279ebf2d05f072e1ab7d747193db6daa432955a28dd3505a85d0a59 Sep 20 02:09:50.102073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.102040 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:50.102182 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.102105 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:50.102182 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.102133 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:09:50.659208 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.658618 2578 patch_prober.go:28] interesting pod/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 20 02:09:50.659208 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.658702 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 20 02:09:50.715162 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.715123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerStarted","Data":"4778d8e3f6111ec662b6af9e4cc8138dd9bdccd225af89072a7091c415fea972"} Sep 20 02:09:50.715381 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.715219 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerStarted","Data":"e1c2ca7c8279ebf2d05f072e1ab7d747193db6daa432955a28dd3505a85d0a59"} Sep 20 02:09:50.750366 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.750285 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podStartSLOduration=1.750258047 podStartE2EDuration="1.750258047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:09:50.733249789 +0000 UTC m=+1042.251565551" watchObservedRunningTime="2026-09-20 02:09:50.750258047 +0000 UTC m=+1042.268573810" Sep 20 02:09:50.798206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:50.798159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:51.662586 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:51.662552 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:51.663664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:51.663644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:09:51.803810 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:51.803784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:52.729557 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:52.729523 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:52.730526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:52.730506 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:09:53.958012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:53.957984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:09:54.066647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066552 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.066647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.066900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.066900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066776 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.066900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066850 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.066900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066892 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066951 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.066963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067029 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6bfq\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067119 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:54.067154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067128 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data" (OuterVolumeSpecName: "istio-data") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:54.067457 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token\") pod \"905e594d-f592-4d39-a953-15563efc240a\" (UID: \"905e594d-f592-4d39-a953-15563efc240a\") " Sep 20 02:09:54.067457 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067277 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:09:54.067797 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067756 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.067797 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067784 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.067928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067798 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.067928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067812 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.067928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.067829 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905e594d-f592-4d39-a953-15563efc240a-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.069080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.069057 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:09:54.069253 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.069232 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq" (OuterVolumeSpecName: "kube-api-access-d6bfq") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "kube-api-access-d6bfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:54.069613 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.069588 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:09:54.069710 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.069613 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token" (OuterVolumeSpecName: "istio-token") pod "905e594d-f592-4d39-a953-15563efc240a" (UID: "905e594d-f592-4d39-a953-15563efc240a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:09:54.169086 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.169048 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905e594d-f592-4d39-a953-15563efc240a-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.169086 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.169079 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6bfq\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-kube-api-access-d6bfq\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.169302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.169097 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905e594d-f592-4d39-a953-15563efc240a-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.169302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.169107 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905e594d-f592-4d39-a953-15563efc240a-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:09:54.518274 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.518234 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:09:54.519005 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.518983 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:09:54.519112 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.519007 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:09:54.519112 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.519034 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:09:54.519112 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.519042 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:09:54.519293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.519144 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:09:54.521940 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.521895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.524162 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.524140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-725649-serving-cert\"" Sep 20 02:09:54.532488 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.532417 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:09:54.674296 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.674258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.674514 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.674322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.674514 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.674390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzlg\" (UniqueName: \"kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.674514 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.674426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.744840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.744793 2578 generic.go:356] "Generic (PLEG): container finished" podID="905e594d-f592-4d39-a953-15563efc240a" containerID="9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7" exitCode=0 Sep 20 02:09:54.746046 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.746017 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" Sep 20 02:09:54.746478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.746424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerDied","Data":"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7"} Sep 20 02:09:54.746575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.746554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc" event={"ID":"905e594d-f592-4d39-a953-15563efc240a","Type":"ContainerDied","Data":"9ad1e85b914e7e690df810e207858a0b88952b50ffbd85eedd48e40a2d0948ef"} Sep 20 02:09:54.746637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.746608 2578 scope.go:117] "RemoveContainer" containerID="9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7" Sep 20 02:09:54.760103 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.760075 2578 scope.go:117] "RemoveContainer" containerID="bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062" Sep 20 02:09:54.770461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.770437 2578 scope.go:117] "RemoveContainer" containerID="9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7" Sep 20 02:09:54.770773 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:54.770751 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7\": container with ID starting with 9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7 not found: ID does not exist" containerID="9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7" Sep 20 02:09:54.770824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.770782 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7"} err="failed to get container status \"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7\": rpc error: code = NotFound desc = could not find container \"9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7\": container with ID starting with 9356384a53892105b65d231188504b52c08b13d70ac2e0a5692bb9912f977be7 not found: ID does not exist" Sep 20 02:09:54.770824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.770799 2578 scope.go:117] "RemoveContainer" containerID="bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062" Sep 20 02:09:54.771068 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:09:54.771045 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062\": container with ID starting with bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062 not found: ID does not exist" containerID="bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062" Sep 20 02:09:54.771147 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.771078 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062"} err="failed to get container status \"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062\": rpc error: code = NotFound desc = could not find container \"bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062\": container with ID starting with bf7342bf020c9bdc969e82b3b47bdb4806f8693d72212192188effb3fd3c2062 not found: ID does not exist" Sep 20 02:09:54.772272 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.772241 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:09:54.775398 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.775371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.775499 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.775431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.775499 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.775477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzlg\" (UniqueName: \"kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.775620 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.775515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.775899 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.775876 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4cfa5a-openshift-default-76bf6fbdb-nbpxc"] Sep 20 02:09:54.776253 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.776191 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.777957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.777936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.778050 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.777987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.785542 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.785518 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzlg\" (UniqueName: \"kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg\") pod \"maas-api-e2e-explicit-725649-65745d5997-gx2jh\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:54.838849 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:54.838813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:55.016548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.016511 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905e594d-f592-4d39-a953-15563efc240a" path="/var/lib/kubelet/pods/905e594d-f592-4d39-a953-15563efc240a/volumes" Sep 20 02:09:55.121070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.121040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:09:55.122702 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:09:55.122669 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578c8f6b_0762_4438_a388_5490a96ad690.slice/crio-4ca9d4b98f2b9945c270f5c9dcaffffbc2f4ba1f4eb2787c2f86806a8cad03f8 WatchSource:0}: Error finding container 4ca9d4b98f2b9945c270f5c9dcaffffbc2f4ba1f4eb2787c2f86806a8cad03f8: Status 404 returned error can't find the container with id 4ca9d4b98f2b9945c270f5c9dcaffffbc2f4ba1f4eb2787c2f86806a8cad03f8 Sep 20 02:09:55.754351 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.754307 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" event={"ID":"578c8f6b-0762-4438-a388-5490a96ad690","Type":"ContainerStarted","Data":"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832"} Sep 20 02:09:55.754541 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.754378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" event={"ID":"578c8f6b-0762-4438-a388-5490a96ad690","Type":"ContainerStarted","Data":"4ca9d4b98f2b9945c270f5c9dcaffffbc2f4ba1f4eb2787c2f86806a8cad03f8"} Sep 20 02:09:55.754541 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.754467 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:09:55.772601 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:09:55.772544 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" podStartSLOduration=1.363896541 podStartE2EDuration="1.772525887s" totalImagesPullingTime="408.629346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:09:55.770324351 +0000 UTC m=+1047.288640105" watchObservedRunningTime="2026-09-20 02:09:55.772525887 +0000 UTC m=+1047.290842046" Sep 20 02:10:00.781287 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:00.781252 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:10:00.781817 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:00.781544 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" podUID="53351b46-7d9a-4523-b649-aec05a0ae5c5" containerName="maas-api" containerID="cri-o://a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe" gracePeriod=30 Sep 20 02:10:01.033429 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.033365 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:10:01.140396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.140360 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs\") pod \"53351b46-7d9a-4523-b649-aec05a0ae5c5\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " Sep 20 02:10:01.140622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.140474 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls\") pod \"53351b46-7d9a-4523-b649-aec05a0ae5c5\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " Sep 20 02:10:01.140622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.140518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca\") pod \"53351b46-7d9a-4523-b649-aec05a0ae5c5\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " Sep 20 02:10:01.140622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.140563 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d84q7\" (UniqueName: \"kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7\") pod \"53351b46-7d9a-4523-b649-aec05a0ae5c5\" (UID: \"53351b46-7d9a-4523-b649-aec05a0ae5c5\") " Sep 20 02:10:01.141818 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.141758 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53351b46-7d9a-4523-b649-aec05a0ae5c5" (UID: "53351b46-7d9a-4523-b649-aec05a0ae5c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:10:01.143775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.143745 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:10:01.144027 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.144002 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" containerID="cri-o://feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9" gracePeriod=30 Sep 20 02:10:01.145480 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.145340 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7" (OuterVolumeSpecName: "kube-api-access-d84q7") pod "53351b46-7d9a-4523-b649-aec05a0ae5c5" (UID: "53351b46-7d9a-4523-b649-aec05a0ae5c5"). InnerVolumeSpecName "kube-api-access-d84q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:10:01.145480 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.145435 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53351b46-7d9a-4523-b649-aec05a0ae5c5" (UID: "53351b46-7d9a-4523-b649-aec05a0ae5c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:10:01.145792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.145752 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53351b46-7d9a-4523-b649-aec05a0ae5c5" (UID: "53351b46-7d9a-4523-b649-aec05a0ae5c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:10:01.242072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.242036 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:01.242072 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.242073 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53351b46-7d9a-4523-b649-aec05a0ae5c5-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:01.242273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.242089 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53351b46-7d9a-4523-b649-aec05a0ae5c5-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:01.242273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.242105 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84q7\" (UniqueName: \"kubernetes.io/projected/53351b46-7d9a-4523-b649-aec05a0ae5c5-kube-api-access-d84q7\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:01.766519 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.766489 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:10:01.796514 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.796473 2578 generic.go:356] "Generic (PLEG): container finished" podID="53351b46-7d9a-4523-b649-aec05a0ae5c5" containerID="a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe" exitCode=0 Sep 20 02:10:01.797012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.796532 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" Sep 20 02:10:01.797012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.796543 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" event={"ID":"53351b46-7d9a-4523-b649-aec05a0ae5c5","Type":"ContainerDied","Data":"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe"} Sep 20 02:10:01.797012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.796591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42" event={"ID":"53351b46-7d9a-4523-b649-aec05a0ae5c5","Type":"ContainerDied","Data":"b105ba03ed738284e2893249fdee7871728faba96e9ec194175f078ad557112f"} Sep 20 02:10:01.797012 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.796620 2578 scope.go:117] "RemoveContainer" containerID="a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe" Sep 20 02:10:01.809206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.809178 2578 scope.go:117] "RemoveContainer" containerID="a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe" Sep 20 02:10:01.809668 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:10:01.809640 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe\": container with ID starting with a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe not found: ID does not exist" containerID="a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe" Sep 20 02:10:01.809799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.809683 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe"} err="failed to get container status \"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe\": rpc error: code = NotFound desc = could not find container \"a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe\": container with ID starting with a49da562bea8804db535010387b0cec8605a626856978a98d142639bcdfbdabe not found: ID does not exist" Sep 20 02:10:01.823071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.823038 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:10:01.829538 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:01.829504 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-066f62f9-7f87fb49cb-gzk42"] Sep 20 02:10:03.016837 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:03.016795 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53351b46-7d9a-4523-b649-aec05a0ae5c5" path="/var/lib/kubelet/pods/53351b46-7d9a-4523-b649-aec05a0ae5c5/volumes" Sep 20 02:10:06.423973 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.423887 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:10:06.490460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490473 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490517 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp6hr\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490626 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490761 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490886 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490784 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:10:06.490886 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490804 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.490886 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.491073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490895 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:10:06.491073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490896 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:10:06.491073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.490928 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket\") pod \"5f26963c-604d-4180-937a-efafdb7a2cee\" (UID: \"5f26963c-604d-4180-937a-efafdb7a2cee\") " Sep 20 02:10:06.491241 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491140 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:10:06.491241 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491216 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:10:06.491647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491611 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f26963c-604d-4180-937a-efafdb7a2cee-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.491647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491636 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.491647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491650 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.491933 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491663 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.491933 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.491676 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.493138 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.493093 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:10:06.493263 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.493166 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:10:06.493263 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.493179 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr" (OuterVolumeSpecName: "kube-api-access-rp6hr") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "kube-api-access-rp6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:10:06.493524 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.493505 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f26963c-604d-4180-937a-efafdb7a2cee" (UID: "5f26963c-604d-4180-937a-efafdb7a2cee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:10:06.592364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.592321 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.592364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.592355 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f26963c-604d-4180-937a-efafdb7a2cee-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.592364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.592364 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6hr\" (UniqueName: \"kubernetes.io/projected/5f26963c-604d-4180-937a-efafdb7a2cee-kube-api-access-rp6hr\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.592364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.592374 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f26963c-604d-4180-937a-efafdb7a2cee-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:10:06.830309 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.830214 2578 generic.go:356] "Generic (PLEG): container finished" podID="5f26963c-604d-4180-937a-efafdb7a2cee" containerID="feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9" exitCode=0 Sep 20 02:10:06.830309 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.830272 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" Sep 20 02:10:06.830512 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.830303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerDied","Data":"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9"} Sep 20 02:10:06.830512 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.830391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd" event={"ID":"5f26963c-604d-4180-937a-efafdb7a2cee","Type":"ContainerDied","Data":"02df7186774a1eaa53de6808abb2b798b0d82e1b3e74c835d795849ad30b2877"} Sep 20 02:10:06.830512 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.830418 2578 scope.go:117] "RemoveContainer" containerID="feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9" Sep 20 02:10:06.843438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.843413 2578 scope.go:117] "RemoveContainer" containerID="20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7" Sep 20 02:10:06.853265 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.853241 2578 scope.go:117] "RemoveContainer" containerID="feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9" Sep 20 02:10:06.853602 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:10:06.853583 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9\": container with ID starting with feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9 not found: ID does not exist" containerID="feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9" Sep 20 02:10:06.853679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.853610 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9"} err="failed to get container status \"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9\": rpc error: code = NotFound desc = could not find container \"feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9\": container with ID starting with feda67ca04ea03065662416eb00c0c1fbe78952a40f8028ed6e29c371f6637a9 not found: ID does not exist" Sep 20 02:10:06.853679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.853628 2578 scope.go:117] "RemoveContainer" containerID="20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7" Sep 20 02:10:06.854017 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:10:06.853950 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7\": container with ID starting with 20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7 not found: ID does not exist" containerID="20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7" Sep 20 02:10:06.854017 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.853990 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7"} err="failed to get container status \"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7\": rpc error: code = NotFound desc = could not find container \"20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7\": container with ID starting with 20cfc391da2a4a22a59a357af8bf52defe2b0da690a4ad2362b093defcd7f0d7 not found: ID does not exist" Sep 20 02:10:06.855039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.855017 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:10:06.858834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:06.858812 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-066f62f9-openshift-default-68cf7b7b84-h7njd"] Sep 20 02:10:07.016875 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.016840 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" path="/var/lib/kubelet/pods/5f26963c-604d-4180-937a-efafdb7a2cee/volumes" Sep 20 02:10:07.799965 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.799927 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:10:07.800354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800334 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800345 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800355 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800368 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800381 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53351b46-7d9a-4523-b649-aec05a0ae5c5" containerName="maas-api" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800386 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53351b46-7d9a-4523-b649-aec05a0ae5c5" containerName="maas-api" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800456 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800465 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f26963c-604d-4180-937a-efafdb7a2cee" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800474 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="905e594d-f592-4d39-a953-15563efc240a" containerName="istio-proxy" Sep 20 02:10:07.800481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.800480 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53351b46-7d9a-4523-b649-aec05a0ae5c5" containerName="maas-api" Sep 20 02:10:07.803325 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.803307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.805718 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.805693 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"premium-w0-8195ba-kserve-self-signed-certs\"" Sep 20 02:10:07.812734 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.812703 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:10:07.903792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4b4\" (UniqueName: \"kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.903994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.903994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.903994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.903994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:07.903994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:07.903957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005411 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4b4\" (UniqueName: \"kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.005827 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.005696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.006036 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.006007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.006036 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.006023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.006196 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.006091 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.008170 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.008147 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.008459 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.008441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.013195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.013174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4b4\" (UniqueName: \"kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4\") pod \"premium-w0-8195ba-kserve-b574b64b8-r9pkf\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.113774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.113676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:08.218519 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.218478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:10:08.389474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.389444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:10:08.390460 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:10:08.390436 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda49db703_24c3_4212_9656_bd0405ecf3ac.slice/crio-bf64bb0900943cde926928b5204735d876c1cf8751a24ade999eb65c6cfc6851 WatchSource:0}: Error finding container bf64bb0900943cde926928b5204735d876c1cf8751a24ade999eb65c6cfc6851: Status 404 returned error can't find the container with id bf64bb0900943cde926928b5204735d876c1cf8751a24ade999eb65c6cfc6851 Sep 20 02:10:08.843981 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.843857 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" event={"ID":"a49db703-24c3-4212-9656-bd0405ecf3ac","Type":"ContainerStarted","Data":"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace"} Sep 20 02:10:08.843981 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.843899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" event={"ID":"a49db703-24c3-4212-9656-bd0405ecf3ac","Type":"ContainerStarted","Data":"bf64bb0900943cde926928b5204735d876c1cf8751a24ade999eb65c6cfc6851"} Sep 20 02:10:08.865582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:08.865526 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" podStartSLOduration=1.865509255 podStartE2EDuration="1.865509255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:10:08.862441345 +0000 UTC m=+1060.380757104" watchObservedRunningTime="2026-09-20 02:10:08.865509255 +0000 UTC m=+1060.383825045" Sep 20 02:10:18.114394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:18.114361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:18.114394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:18.114404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:18.127447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:18.127421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:18.924033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:18.924002 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:10:19.920029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.919993 2578 generic.go:356] "Generic (PLEG): container finished" podID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerID="4778d8e3f6111ec662b6af9e4cc8138dd9bdccd225af89072a7091c415fea972" exitCode=0 Sep 20 02:10:19.920518 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.920072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerDied","Data":"4778d8e3f6111ec662b6af9e4cc8138dd9bdccd225af89072a7091c415fea972"} Sep 20 02:10:19.920601 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.920583 2578 scope.go:117] "RemoveContainer" containerID="4778d8e3f6111ec662b6af9e4cc8138dd9bdccd225af89072a7091c415fea972" Sep 20 02:10:19.921746 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.921722 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:10:19.921817 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.921785 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:10:19.921854 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:19.921818 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:10:20.928668 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:20.928632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerStarted","Data":"831ddea98f6fbe422b97ebd44258df549ad712586666716c8c345699f4866564"} Sep 20 02:10:21.798360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:21.798317 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:10:21.799549 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:21.799519 2578 patch_prober.go:28] interesting pod/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 20 02:10:21.799696 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:21.799584 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 20 02:10:22.729734 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:22.729697 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:10:22.798727 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:22.798691 2578 patch_prober.go:28] interesting pod/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 20 02:10:22.798943 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:22.798747 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 20 02:10:23.798313 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:23.798273 2578 patch_prober.go:28] interesting pod/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 20 02:10:23.798729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:23.798342 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 20 02:10:24.798589 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:24.798548 2578 patch_prober.go:28] interesting pod/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 20 02:10:24.799066 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:24.798606 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 20 02:10:25.313738 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:25.313698 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:10:25.798166 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:25.798118 2578 patch_prober.go:28] interesting pod/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 20 02:10:25.798336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:25.798188 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 20 02:10:26.803004 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.802953 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:10:26.804026 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.804006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:10:26.914629 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.914597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:10:26.917518 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.917494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:26.919500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.919476 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b15051\"/\"premium-w4-b15051-kserve-self-signed-certs\"" Sep 20 02:10:26.930301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:26.930276 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:10:27.013080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkx9v\" (UniqueName: \"kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.013275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.013275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.013275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.013275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.013275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.013258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114580 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkx9v\" (UniqueName: \"kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.114996 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.114993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.115155 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.115080 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.115372 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.115350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.117842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.117819 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.118138 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.118119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.123162 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.123139 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkx9v\" (UniqueName: \"kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v\") pod \"premium-w4-b15051-kserve-7fd996d886-wjlwh\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.231138 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.231104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:27.507372 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.507339 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:10:27.508367 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:10:27.508345 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeba8cd0c_6a62_4c78_8506_4aea061aad6a.slice/crio-91aa9e6de361475a1b98485864dbc2aea5d00f9a0ff98c704a3ff170864dfe41 WatchSource:0}: Error finding container 91aa9e6de361475a1b98485864dbc2aea5d00f9a0ff98c704a3ff170864dfe41: Status 404 returned error can't find the container with id 91aa9e6de361475a1b98485864dbc2aea5d00f9a0ff98c704a3ff170864dfe41 Sep 20 02:10:27.983526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.983489 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" event={"ID":"eba8cd0c-6a62-4c78-8506-4aea061aad6a","Type":"ContainerStarted","Data":"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926"} Sep 20 02:10:27.983956 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:27.983540 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" event={"ID":"eba8cd0c-6a62-4c78-8506-4aea061aad6a","Type":"ContainerStarted","Data":"91aa9e6de361475a1b98485864dbc2aea5d00f9a0ff98c704a3ff170864dfe41"} Sep 20 02:10:28.001884 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:28.001830 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" podStartSLOduration=2.00181331 podStartE2EDuration="2.00181331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:10:27.998977147 +0000 UTC m=+1079.517292905" watchObservedRunningTime="2026-09-20 02:10:28.00181331 +0000 UTC m=+1079.520129101" Sep 20 02:10:35.998926 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:35.998860 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:10:36.002290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.002271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.004561 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.004535 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-703998\"/\"openshift-service-ca.crt\"" Sep 20 02:10:36.005297 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.005268 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-703998\"/\"test-model-703998-kserve-self-signed-certs\"" Sep 20 02:10:36.005297 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.005280 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-703998\"/\"kube-root-ca.crt\"" Sep 20 02:10:36.005477 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.005445 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-703998\"/\"default-dockercfg-txknt\"" Sep 20 02:10:36.012629 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.012601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:10:36.106154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4tf\" (UniqueName: \"kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.106358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.106358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.106358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.106358 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.106542 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.106394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207501 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4tf\" (UniqueName: \"kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207703 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.207953 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.208022 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.207996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.208078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.208051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.208133 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.208095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.210083 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.210055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.210314 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.210295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.215553 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.215531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4tf\" (UniqueName: \"kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf\") pod \"test-model-703998-kserve-b6794ccbf-6qq7w\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.313229 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.313124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:36.592673 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:36.592640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:10:36.594434 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:10:36.594407 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96a58d5_6155_41ba_86dc_6d35789d001c.slice/crio-bb8877e253902789911c5fe8a7bb457886689f85bc69d8c34a05f688ebccf7ea WatchSource:0}: Error finding container bb8877e253902789911c5fe8a7bb457886689f85bc69d8c34a05f688ebccf7ea: Status 404 returned error can't find the container with id bb8877e253902789911c5fe8a7bb457886689f85bc69d8c34a05f688ebccf7ea Sep 20 02:10:37.036959 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.036901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" event={"ID":"a96a58d5-6155-41ba-86dc-6d35789d001c","Type":"ContainerStarted","Data":"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20"} Sep 20 02:10:37.037356 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.036973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" event={"ID":"a96a58d5-6155-41ba-86dc-6d35789d001c","Type":"ContainerStarted","Data":"bb8877e253902789911c5fe8a7bb457886689f85bc69d8c34a05f688ebccf7ea"} Sep 20 02:10:37.054727 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.054672 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" podStartSLOduration=2.054651903 podStartE2EDuration="2.054651903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:10:37.053209357 +0000 UTC m=+1088.571525114" watchObservedRunningTime="2026-09-20 02:10:37.054651903 +0000 UTC m=+1088.572967661" Sep 20 02:10:37.232093 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.232061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:37.232093 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.232106 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:37.244894 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:37.244868 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:38.055003 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:38.054971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:10:44.915414 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:44.915372 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:10:46.313816 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:46.313770 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:46.314247 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:46.313841 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:46.329295 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:46.329262 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:10:47.106335 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:10:47.106305 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:11:11.026348 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:11.026311 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:11:22.274786 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.274751 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:11:22.275409 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.275007 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" podUID="578c8f6b-0762-4438-a388-5490a96ad690" containerName="maas-api" containerID="cri-o://6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832" gracePeriod=30 Sep 20 02:11:22.527860 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.527791 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:11:22.688775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.688739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs\") pod \"578c8f6b-0762-4438-a388-5490a96ad690\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " Sep 20 02:11:22.688775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.688779 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca\") pod \"578c8f6b-0762-4438-a388-5490a96ad690\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " Sep 20 02:11:22.689122 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.688818 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls\") pod \"578c8f6b-0762-4438-a388-5490a96ad690\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " Sep 20 02:11:22.689122 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.688974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrzlg\" (UniqueName: \"kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg\") pod \"578c8f6b-0762-4438-a388-5490a96ad690\" (UID: \"578c8f6b-0762-4438-a388-5490a96ad690\") " Sep 20 02:11:22.689242 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.689196 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "578c8f6b-0762-4438-a388-5490a96ad690" (UID: "578c8f6b-0762-4438-a388-5490a96ad690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:11:22.689367 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.689344 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578c8f6b-0762-4438-a388-5490a96ad690-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:22.691469 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.691427 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "578c8f6b-0762-4438-a388-5490a96ad690" (UID: "578c8f6b-0762-4438-a388-5490a96ad690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:11:22.692275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.692250 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "578c8f6b-0762-4438-a388-5490a96ad690" (UID: "578c8f6b-0762-4438-a388-5490a96ad690"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:11:22.692371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.692278 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg" (OuterVolumeSpecName: "kube-api-access-lrzlg") pod "578c8f6b-0762-4438-a388-5490a96ad690" (UID: "578c8f6b-0762-4438-a388-5490a96ad690"). InnerVolumeSpecName "kube-api-access-lrzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:11:22.790678 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.790574 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrzlg\" (UniqueName: \"kubernetes.io/projected/578c8f6b-0762-4438-a388-5490a96ad690-kube-api-access-lrzlg\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:22.790678 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.790611 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:22.790678 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:22.790627 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578c8f6b-0762-4438-a388-5490a96ad690-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:23.319824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.319789 2578 generic.go:356] "Generic (PLEG): container finished" podID="578c8f6b-0762-4438-a388-5490a96ad690" containerID="6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832" exitCode=0 Sep 20 02:11:23.320301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.319835 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" Sep 20 02:11:23.320301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.319865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" event={"ID":"578c8f6b-0762-4438-a388-5490a96ad690","Type":"ContainerDied","Data":"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832"} Sep 20 02:11:23.320301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.319941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh" event={"ID":"578c8f6b-0762-4438-a388-5490a96ad690","Type":"ContainerDied","Data":"4ca9d4b98f2b9945c270f5c9dcaffffbc2f4ba1f4eb2787c2f86806a8cad03f8"} Sep 20 02:11:23.320301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.319966 2578 scope.go:117] "RemoveContainer" containerID="6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832" Sep 20 02:11:23.330349 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.330326 2578 scope.go:117] "RemoveContainer" containerID="6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832" Sep 20 02:11:23.330664 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:11:23.330636 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832\": container with ID starting with 6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832 not found: ID does not exist" containerID="6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832" Sep 20 02:11:23.330717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.330667 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832"} err="failed to get container status \"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832\": rpc error: code = NotFound desc = could not find container \"6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832\": container with ID starting with 6e967437fbbce20923037a43e22c24d2d1a0f644650fdce1d12462ad55591832 not found: ID does not exist" Sep 20 02:11:23.337314 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.337283 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:11:23.340858 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:23.340833 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-725649-65745d5997-gx2jh"] Sep 20 02:11:25.016687 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:25.016651 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="578c8f6b-0762-4438-a388-5490a96ad690" path="/var/lib/kubelet/pods/578c8f6b-0762-4438-a388-5490a96ad690/volumes" Sep 20 02:11:29.163264 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.163229 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:11:29.163753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.163484 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" containerID="cri-o://831ddea98f6fbe422b97ebd44258df549ad712586666716c8c345699f4866564" gracePeriod=30 Sep 20 02:11:29.956103 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.956072 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:11:29.956462 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.956449 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="578c8f6b-0762-4438-a388-5490a96ad690" containerName="maas-api" Sep 20 02:11:29.956523 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.956465 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="578c8f6b-0762-4438-a388-5490a96ad690" containerName="maas-api" Sep 20 02:11:29.956578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.956549 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="578c8f6b-0762-4438-a388-5490a96ad690" containerName="maas-api" Sep 20 02:11:29.959253 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.959236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:29.961563 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.961541 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1750e4-openshift-default-dockercfg-5n9gj\"" Sep 20 02:11:29.975389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:29.975356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:11:30.058374 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058785 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.058867 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.058850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.059225 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.059191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6df\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.059330 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.059236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.160818 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.160777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.160837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.160892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6df\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.160947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161055 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161329 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161329 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161329 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161616 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161505 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161614 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.161847 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.161829 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.163798 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.163771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.164136 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.163992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.171330 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.171278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6df\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.171494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.171469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token\") pod \"e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.270304 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.270216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:30.546796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.546764 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:11:30.548491 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:11:30.548459 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b9a862_b4dd_424a_a291_94fca0452311.slice/crio-be37aeab941813f1789d4904891db27416576d37d4043badf8ae4e79d018e311 WatchSource:0}: Error finding container be37aeab941813f1789d4904891db27416576d37d4043badf8ae4e79d018e311: Status 404 returned error can't find the container with id be37aeab941813f1789d4904891db27416576d37d4043badf8ae4e79d018e311 Sep 20 02:11:30.551772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.551734 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:30.551868 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.551814 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:30.551943 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:30.551871 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:31.375374 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:31.375339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" event={"ID":"b2b9a862-b4dd-424a-a291-94fca0452311","Type":"ContainerStarted","Data":"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee"} Sep 20 02:11:31.375769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:31.375402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" event={"ID":"b2b9a862-b4dd-424a-a291-94fca0452311","Type":"ContainerStarted","Data":"be37aeab941813f1789d4904891db27416576d37d4043badf8ae4e79d018e311"} Sep 20 02:11:31.396107 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:31.396061 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" podStartSLOduration=2.396045478 podStartE2EDuration="2.396045478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:11:31.394199469 +0000 UTC m=+1142.912515269" watchObservedRunningTime="2026-09-20 02:11:31.396045478 +0000 UTC m=+1142.914361236" Sep 20 02:11:32.270421 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:32.270378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:32.276366 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:32.276339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:32.381382 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:32.381347 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:32.382394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:32.382373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:11:33.114428 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:33.114395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:11:34.395329 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.395294 2578 generic.go:356] "Generic (PLEG): container finished" podID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerID="831ddea98f6fbe422b97ebd44258df549ad712586666716c8c345699f4866564" exitCode=0 Sep 20 02:11:34.395658 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.395365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerDied","Data":"831ddea98f6fbe422b97ebd44258df549ad712586666716c8c345699f4866564"} Sep 20 02:11:34.395658 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.395425 2578 scope.go:117] "RemoveContainer" containerID="4778d8e3f6111ec662b6af9e4cc8138dd9bdccd225af89072a7091c415fea972" Sep 20 02:11:34.418654 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.418635 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:11:34.501706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501623 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.501706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.501706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501701 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501727 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501756 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501773 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501792 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501826 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.501855 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xpmk\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk\") pod \"eeee7480-4894-4886-a9fd-1f8c76bf3774\" (UID: \"eeee7480-4894-4886-a9fd-1f8c76bf3774\") " Sep 20 02:11:34.502023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502002 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:11:34.502394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502142 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:11:34.502700 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502169 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data" (OuterVolumeSpecName: "istio-data") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:11:34.502769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502340 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:11:34.502769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:11:34.502769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502402 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.502769 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.502740 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.504124 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.504101 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:11:34.504494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.504469 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk" (OuterVolumeSpecName: "kube-api-access-9xpmk") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "kube-api-access-9xpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:11:34.504688 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.504671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token" (OuterVolumeSpecName: "istio-token") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:11:34.504688 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.504673 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eeee7480-4894-4886-a9fd-1f8c76bf3774" (UID: "eeee7480-4894-4886-a9fd-1f8c76bf3774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:11:34.604187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604152 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604181 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604192 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604204 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604214 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeee7480-4894-4886-a9fd-1f8c76bf3774-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604226 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xpmk\" (UniqueName: \"kubernetes.io/projected/eeee7480-4894-4886-a9fd-1f8c76bf3774-kube-api-access-9xpmk\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:34.604416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:34.604239 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeee7480-4894-4886-a9fd-1f8c76bf3774-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:11:35.403209 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:35.403176 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" Sep 20 02:11:35.403622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:35.403187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2" event={"ID":"eeee7480-4894-4886-a9fd-1f8c76bf3774","Type":"ContainerDied","Data":"e1c2ca7c8279ebf2d05f072e1ab7d747193db6daa432955a28dd3505a85d0a59"} Sep 20 02:11:35.403622 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:35.403314 2578 scope.go:117] "RemoveContainer" containerID="831ddea98f6fbe422b97ebd44258df549ad712586666716c8c345699f4866564" Sep 20 02:11:35.441522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:35.441484 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:11:35.446889 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:35.446853 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-725649-openshift-default-7b5b9999b4-6j6h2"] Sep 20 02:11:37.015500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:37.015467 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" path="/var/lib/kubelet/pods/eeee7480-4894-4886-a9fd-1f8c76bf3774/volumes" Sep 20 02:11:38.317028 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.316988 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:11:38.317493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.317413 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:38.317493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.317425 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:38.317598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.317511 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:38.317598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.317524 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:38.321727 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.321704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.323987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.323958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06367b84-openshift-default-dockercfg-69v7v\"" Sep 20 02:11:38.335584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.335558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:11:38.444063 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpl6r\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.444466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.444434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545377 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpl6r\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.545986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.545666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.546190 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.546162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.546274 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.546172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.546400 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.546307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.546461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.546408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.546522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.546502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.548216 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.548198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.548403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.548388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.553775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.553751 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.554036 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.554016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpl6r\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r\") pod \"e2e-mt-06367b84-openshift-default-69d5cc7568-448dr\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.634859 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.634767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:38.999199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:38.999170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:11:39.001378 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:11:39.001351 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845914af_9ab6_4005_b2db_0c60e89dcf42.slice/crio-68447074ab68f95260434cc9632208fe9e94cc2e35dc1a94d32dfa81db45d98d WatchSource:0}: Error finding container 68447074ab68f95260434cc9632208fe9e94cc2e35dc1a94d32dfa81db45d98d: Status 404 returned error can't find the container with id 68447074ab68f95260434cc9632208fe9e94cc2e35dc1a94d32dfa81db45d98d Sep 20 02:11:39.003501 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.003473 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:39.003580 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.003540 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:39.003580 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.003567 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:11:39.432250 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.432215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerStarted","Data":"f32d88a38aeb028f271ebba3be7f2cd5e85475236783caa5237269b32f520205"} Sep 20 02:11:39.432660 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.432274 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerStarted","Data":"68447074ab68f95260434cc9632208fe9e94cc2e35dc1a94d32dfa81db45d98d"} Sep 20 02:11:39.453344 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.453292 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podStartSLOduration=1.45327236 podStartE2EDuration="1.45327236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:11:39.452176325 +0000 UTC m=+1150.970492098" watchObservedRunningTime="2026-09-20 02:11:39.45327236 +0000 UTC m=+1150.971588120" Sep 20 02:11:39.635670 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:39.635635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:40.640827 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:40.640797 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:41.444853 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:41.444826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:41.445969 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:41.445951 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:11:47.711128 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.711094 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:11:47.711983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.711544 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:47.711983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.711574 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeee7480-4894-4886-a9fd-1f8c76bf3774" containerName="istio-proxy" Sep 20 02:11:47.714452 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.714433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.716662 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.716639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 20 02:11:47.716781 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.716643 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 20 02:11:47.717364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.717339 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 20 02:11:47.717477 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.717342 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06367b84-serving-cert\"" Sep 20 02:11:47.717477 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.717437 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z4spz\"" Sep 20 02:11:47.725494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.725462 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:11:47.842511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.842469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.842708 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.842563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcj7\" (UniqueName: \"kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.842708 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.842595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.842813 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.842702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.944301 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.944264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.944483 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.944332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.944483 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.944385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcj7\" (UniqueName: \"kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.944483 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.944413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.945125 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.945100 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.947126 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.947106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.947258 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.947238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:47.952877 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:47.952851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcj7\" (UniqueName: \"kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7\") pod \"maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:48.031176 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:48.031092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:48.322118 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:48.322089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:11:48.324029 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:11:48.323997 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb9a185_ef7b_4f0e_86f3_f6361affde69.slice/crio-803ca4c358bb0ddd6199492e0d07a554ccbe91edb97148dd4937d32aec508d5e WatchSource:0}: Error finding container 803ca4c358bb0ddd6199492e0d07a554ccbe91edb97148dd4937d32aec508d5e: Status 404 returned error can't find the container with id 803ca4c358bb0ddd6199492e0d07a554ccbe91edb97148dd4937d32aec508d5e Sep 20 02:11:48.487739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:48.487704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" event={"ID":"7bb9a185-ef7b-4f0e-86f3-f6361affde69","Type":"ContainerStarted","Data":"803ca4c358bb0ddd6199492e0d07a554ccbe91edb97148dd4937d32aec508d5e"} Sep 20 02:11:49.494936 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:49.494885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" event={"ID":"7bb9a185-ef7b-4f0e-86f3-f6361affde69","Type":"ContainerStarted","Data":"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378"} Sep 20 02:11:49.495322 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:49.495025 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:49.513835 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:49.513775 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" podStartSLOduration=2.082018651 podStartE2EDuration="2.513758376s" totalImagesPullingTime="431.739725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:11:49.511305624 +0000 UTC m=+1161.029621382" watchObservedRunningTime="2026-09-20 02:11:49.513758376 +0000 UTC m=+1161.032074134" Sep 20 02:11:55.506453 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:55.506374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:11:56.921298 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:11:56.921264 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:12:07.620187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.620087 2578 generic.go:356] "Generic (PLEG): container finished" podID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerID="f32d88a38aeb028f271ebba3be7f2cd5e85475236783caa5237269b32f520205" exitCode=0 Sep 20 02:12:07.620187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.620165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerDied","Data":"f32d88a38aeb028f271ebba3be7f2cd5e85475236783caa5237269b32f520205"} Sep 20 02:12:07.620711 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.620566 2578 scope.go:117] "RemoveContainer" containerID="f32d88a38aeb028f271ebba3be7f2cd5e85475236783caa5237269b32f520205" Sep 20 02:12:07.621798 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.621769 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:07.621864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.621839 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:07.621901 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:07.621870 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:08.628420 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:08.628388 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerStarted","Data":"73a9fbff243ab22088889d31903380ec20a4c7a23a4f424477b5eca2c65e9480"} Sep 20 02:12:08.635421 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:08.635397 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:08.636590 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:08.636567 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:08.636696 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:08.636616 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:09.636625 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:09.636577 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:09.637154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:09.636643 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:10.635948 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:10.635876 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:10.636167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:10.635976 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:11.445120 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:11.445083 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:11.636281 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:11.636240 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:11.636552 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:11.636324 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:12.645407 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:12.645365 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:12.645936 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:12.645455 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:13.636091 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:13.636053 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:13.636408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:13.636379 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:14.635795 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:14.635744 2578 patch_prober.go:28] interesting pod/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 20 02:12:14.636317 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:14.635831 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 20 02:12:15.640415 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:15.640385 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:15.641515 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:15.641496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:21.329125 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:21.329094 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:12:29.072326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:29.072293 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:12:29.103003 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:29.102976 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:12:32.188829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.188797 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:12:32.189370 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.189102 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" podUID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" containerName="maas-api" containerID="cri-o://ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378" gracePeriod=30 Sep 20 02:12:32.429193 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.429166 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:12:32.599153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca\") pod \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " Sep 20 02:12:32.599153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599148 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkcj7\" (UniqueName: \"kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7\") pod \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " Sep 20 02:12:32.599382 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs\") pod \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " Sep 20 02:12:32.599382 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599226 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls\") pod \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\" (UID: \"7bb9a185-ef7b-4f0e-86f3-f6361affde69\") " Sep 20 02:12:32.599505 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599460 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bb9a185-ef7b-4f0e-86f3-f6361affde69" (UID: "7bb9a185-ef7b-4f0e-86f3-f6361affde69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:12:32.599670 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.599646 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9a185-ef7b-4f0e-86f3-f6361affde69-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:32.601577 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.601549 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bb9a185-ef7b-4f0e-86f3-f6361affde69" (UID: "7bb9a185-ef7b-4f0e-86f3-f6361affde69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:12:32.601964 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.601931 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bb9a185-ef7b-4f0e-86f3-f6361affde69" (UID: "7bb9a185-ef7b-4f0e-86f3-f6361affde69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:12:32.601964 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.601951 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7" (OuterVolumeSpecName: "kube-api-access-rkcj7") pod "7bb9a185-ef7b-4f0e-86f3-f6361affde69" (UID: "7bb9a185-ef7b-4f0e-86f3-f6361affde69"). InnerVolumeSpecName "kube-api-access-rkcj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:12:32.700460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.700417 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcj7\" (UniqueName: \"kubernetes.io/projected/7bb9a185-ef7b-4f0e-86f3-f6361affde69-kube-api-access-rkcj7\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:32.700460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.700460 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:32.700676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.700474 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9a185-ef7b-4f0e-86f3-f6361affde69-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:32.784740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.784705 2578 generic.go:356] "Generic (PLEG): container finished" podID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" containerID="ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378" exitCode=0 Sep 20 02:12:32.784896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.784754 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" Sep 20 02:12:32.784896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.784786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" event={"ID":"7bb9a185-ef7b-4f0e-86f3-f6361affde69","Type":"ContainerDied","Data":"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378"} Sep 20 02:12:32.784896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.784847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl" event={"ID":"7bb9a185-ef7b-4f0e-86f3-f6361affde69","Type":"ContainerDied","Data":"803ca4c358bb0ddd6199492e0d07a554ccbe91edb97148dd4937d32aec508d5e"} Sep 20 02:12:32.784896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.784876 2578 scope.go:117] "RemoveContainer" containerID="ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378" Sep 20 02:12:32.794822 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.794803 2578 scope.go:117] "RemoveContainer" containerID="ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378" Sep 20 02:12:32.795130 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:12:32.795110 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378\": container with ID starting with ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378 not found: ID does not exist" containerID="ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378" Sep 20 02:12:32.795221 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.795137 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378"} err="failed to get container status \"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378\": rpc error: code = NotFound desc = could not find container \"ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378\": container with ID starting with ec3662cd0b0a6f548214c1634b520efe6c4da6fb4614d66e94f1e3273695f378 not found: ID does not exist" Sep 20 02:12:32.806441 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.806411 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:12:32.810010 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:32.809984 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06367b84-6fbc8bdfb5-75tpl"] Sep 20 02:12:33.016194 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:33.016161 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" path="/var/lib/kubelet/pods/7bb9a185-ef7b-4f0e-86f3-f6361affde69/volumes" Sep 20 02:12:38.638310 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:38.638275 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:12:38.638761 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:38.638498 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" containerID="cri-o://73a9fbff243ab22088889d31903380ec20a4c7a23a4f424477b5eca2c65e9480" gracePeriod=30 Sep 20 02:12:43.852458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:43.852418 2578 generic.go:356] "Generic (PLEG): container finished" podID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerID="73a9fbff243ab22088889d31903380ec20a4c7a23a4f424477b5eca2c65e9480" exitCode=0 Sep 20 02:12:43.852930 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:43.852486 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerDied","Data":"73a9fbff243ab22088889d31903380ec20a4c7a23a4f424477b5eca2c65e9480"} Sep 20 02:12:43.852930 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:43.852554 2578 scope.go:117] "RemoveContainer" containerID="f32d88a38aeb028f271ebba3be7f2cd5e85475236783caa5237269b32f520205" Sep 20 02:12:43.892475 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:43.892448 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:44.012492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012499 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpl6r\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012532 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012558 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012629 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert\") pod \"845914af-9ab6-4005-b2db-0c60e89dcf42\" (UID: \"845914af-9ab6-4005-b2db-0c60e89dcf42\") " Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012742 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data" (OuterVolumeSpecName: "istio-data") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:12:44.012791 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012750 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012815 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.012942 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.013065 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.013084 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.013097 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.013108 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.013246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.013120 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:12:44.014985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.014945 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:12:44.014985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.014962 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r" (OuterVolumeSpecName: "kube-api-access-hpl6r") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "kube-api-access-hpl6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:12:44.015126 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.015035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:12:44.015126 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.015060 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token" (OuterVolumeSpecName: "istio-token") pod "845914af-9ab6-4005-b2db-0c60e89dcf42" (UID: "845914af-9ab6-4005-b2db-0c60e89dcf42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:12:44.114349 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.114319 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845914af-9ab6-4005-b2db-0c60e89dcf42-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.114349 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.114341 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpl6r\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-kube-api-access-hpl6r\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.114349 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.114352 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.114555 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.114360 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.114555 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.114368 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845914af-9ab6-4005-b2db-0c60e89dcf42-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:12:44.859326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.859285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" event={"ID":"845914af-9ab6-4005-b2db-0c60e89dcf42","Type":"ContainerDied","Data":"68447074ab68f95260434cc9632208fe9e94cc2e35dc1a94d32dfa81db45d98d"} Sep 20 02:12:44.859326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.859307 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr" Sep 20 02:12:44.859841 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.859346 2578 scope.go:117] "RemoveContainer" containerID="73a9fbff243ab22088889d31903380ec20a4c7a23a4f424477b5eca2c65e9480" Sep 20 02:12:44.883056 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.883025 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:12:44.887232 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:44.887201 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06367b84-openshift-default-69d5cc7568-448dr"] Sep 20 02:12:45.015475 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:45.015440 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" path="/var/lib/kubelet/pods/845914af-9ab6-4005-b2db-0c60e89dcf42/volumes" Sep 20 02:12:52.239312 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239277 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239708 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" containerName="maas-api" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239720 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" containerName="maas-api" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239734 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239739 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239747 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239752 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239832 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb9a185-ef7b-4f0e-86f3-f6361affde69" containerName="maas-api" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239841 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.239892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.239850 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="845914af-9ab6-4005-b2db-0c60e89dcf42" containerName="istio-proxy" Sep 20 02:12:52.242743 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.242723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.245128 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.245106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4559b3-openshift-default-dockercfg-xvq5p\"" Sep 20 02:12:52.255960 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.255929 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:12:52.295158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxkk\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295556 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295556 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.295556 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.295477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxkk\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.396869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.397228 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.396870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.397228 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.397199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.398871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.398845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.399020 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.399004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.403992 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.403965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxkk\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.403992 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.403979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token\") pod \"e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.555078 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.554976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:52.830737 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.830710 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:12:52.832324 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:12:52.832296 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb476887d_ae63_491c_9d46_4ab4cf2c67cf.slice/crio-4dc94f6f532eb7d6f62c712d22a2604f32e1aa8ad1af3d6cf899ee400aaa526e WatchSource:0}: Error finding container 4dc94f6f532eb7d6f62c712d22a2604f32e1aa8ad1af3d6cf899ee400aaa526e: Status 404 returned error can't find the container with id 4dc94f6f532eb7d6f62c712d22a2604f32e1aa8ad1af3d6cf899ee400aaa526e Sep 20 02:12:52.835169 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.835133 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:52.835260 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.835209 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:52.835260 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.835239 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:12:52.917302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.917268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerStarted","Data":"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f"} Sep 20 02:12:52.917485 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.917326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerStarted","Data":"4dc94f6f532eb7d6f62c712d22a2604f32e1aa8ad1af3d6cf899ee400aaa526e"} Sep 20 02:12:52.940615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:52.940552 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podStartSLOduration=0.94053325 podStartE2EDuration="940.53325ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:12:52.937568236 +0000 UTC m=+1224.455883998" watchObservedRunningTime="2026-09-20 02:12:52.94053325 +0000 UTC m=+1224.458849008" Sep 20 02:12:53.556059 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:53.556023 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:53.561885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:53.561853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:53.923442 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:53.923416 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:53.924416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:53.924400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:12:56.221246 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:56.221205 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:12:58.114589 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.114552 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:12:58.117049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.117030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.119174 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.119148 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4559b3-serving-cert\"" Sep 20 02:12:58.120131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.120098 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z4spz\"" Sep 20 02:12:58.120264 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.120100 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 20 02:12:58.120264 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.120100 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 20 02:12:58.120420 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.120399 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 20 02:12:58.130042 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.130018 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:12:58.262681 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.262650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.262871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.262712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.262871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.262765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fh58\" (UniqueName: \"kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.262871 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.262816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.363651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.363622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.363821 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.363677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.363821 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.363728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.363821 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.363753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fh58\" (UniqueName: \"kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.364293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.364268 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.366176 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.366120 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.366305 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.366289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.373149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.373128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fh58\" (UniqueName: \"kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58\") pod \"maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.432757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.432728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:12:58.709272 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.709240 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:12:58.710195 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:12:58.710169 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d643321_8b1f_4c40_a1c3_7b1e5f66b0f3.slice/crio-9791572fff88d5eefaff648b7ad12535f5a4b6f68b422f4051648763e95cb84e WatchSource:0}: Error finding container 9791572fff88d5eefaff648b7ad12535f5a4b6f68b422f4051648763e95cb84e: Status 404 returned error can't find the container with id 9791572fff88d5eefaff648b7ad12535f5a4b6f68b422f4051648763e95cb84e Sep 20 02:12:58.964142 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:58.964052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" event={"ID":"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3","Type":"ContainerStarted","Data":"9791572fff88d5eefaff648b7ad12535f5a4b6f68b422f4051648763e95cb84e"} Sep 20 02:12:59.971484 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:59.971444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" event={"ID":"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3","Type":"ContainerStarted","Data":"b01dbd07287b305687cf275b1a7e95a0fafd79cb9e2c17315dd66b56527b2dd4"} Sep 20 02:12:59.971971 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:12:59.971527 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:13:00.011655 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.011602 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" podStartSLOduration=1.683990808 podStartE2EDuration="2.011587637s" totalImagesPullingTime="327.596829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:13:00.008983063 +0000 UTC m=+1231.527298821" watchObservedRunningTime="2026-09-20 02:13:00.011587637 +0000 UTC m=+1231.529903420" Sep 20 02:13:00.156003 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.155969 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:13:00.156244 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.156219 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" containerName="istio-proxy" containerID="cri-o://d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee" gracePeriod=30 Sep 20 02:13:00.599926 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.599877 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:13:00.602429 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.602406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.604766 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.604745 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8195ba\"/\"distinct-w0-8195ba-kserve-self-signed-certs\"" Sep 20 02:13:00.616179 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.616155 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:13:00.687045 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.687187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.687187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.687187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.687187 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqkg\" (UniqueName: \"kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.687326 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.687216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788211 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqkg\" (UniqueName: \"kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788379 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788544 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788544 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788688 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788669 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788764 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788723 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.788764 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.788737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.790716 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.790695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.791023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.791005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.800572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.800550 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqkg\" (UniqueName: \"kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg\") pod \"distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:00.912461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:00.912422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:01.187115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:01.187090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:13:01.187885 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:13:01.187861 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c747e1_1d4d_4184_b5e4_b0206073eff1.slice/crio-a40bc272e1da30357ccff75c5b3a91db580c8a2c28e742583d3b56d7d6f6697c WatchSource:0}: Error finding container a40bc272e1da30357ccff75c5b3a91db580c8a2c28e742583d3b56d7d6f6697c: Status 404 returned error can't find the container with id a40bc272e1da30357ccff75c5b3a91db580c8a2c28e742583d3b56d7d6f6697c Sep 20 02:13:01.985280 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:01.985244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" event={"ID":"f3c747e1-1d4d-4184-b5e4-b0206073eff1","Type":"ContainerStarted","Data":"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a"} Sep 20 02:13:01.985280 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:01.985286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" event={"ID":"f3c747e1-1d4d-4184-b5e4-b0206073eff1","Type":"ContainerStarted","Data":"a40bc272e1da30357ccff75c5b3a91db580c8a2c28e742583d3b56d7d6f6697c"} Sep 20 02:13:02.002303 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:02.002256 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" podStartSLOduration=2.002242195 podStartE2EDuration="2.002242195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:13:02.000492613 +0000 UTC m=+1233.518808396" watchObservedRunningTime="2026-09-20 02:13:02.002242195 +0000 UTC m=+1233.520557953" Sep 20 02:13:02.384479 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:02.384390 2578 patch_prober.go:28] interesting pod/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 20 02:13:02.384479 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:02.384463 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 20 02:13:05.411168 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.411141 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:13:05.545431 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545351 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545431 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545431 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545405 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545487 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj6df\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545644 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:13:05.545704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545653 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.546073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545725 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert\") pod \"b2b9a862-b4dd-424a-a291-94fca0452311\" (UID: \"b2b9a862-b4dd-424a-a291-94fca0452311\") " Sep 20 02:13:05.546073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545811 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:13:05.546073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.545858 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:13:05.546240 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546051 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:13:05.546403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:13:05.546403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546325 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.546403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546392 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.546403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546406 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.546403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.546418 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.547763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.547733 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:13:05.547891 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.547826 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:13:05.547993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.547981 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df" (OuterVolumeSpecName: "kube-api-access-zj6df") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "kube-api-access-zj6df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:13:05.548051 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.547999 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2b9a862-b4dd-424a-a291-94fca0452311" (UID: "b2b9a862-b4dd-424a-a291-94fca0452311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:13:05.647649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.647610 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj6df\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-kube-api-access-zj6df\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.647649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.647641 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b9a862-b4dd-424a-a291-94fca0452311-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.647649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.647655 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b9a862-b4dd-424a-a291-94fca0452311-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.647897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.647668 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b9a862-b4dd-424a-a291-94fca0452311-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.647897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.647680 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b9a862-b4dd-424a-a291-94fca0452311-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:13:05.984346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:05.984312 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:13:06.009564 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.009530 2578 generic.go:356] "Generic (PLEG): container finished" podID="b2b9a862-b4dd-424a-a291-94fca0452311" containerID="d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee" exitCode=0 Sep 20 02:13:06.009741 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.009618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" event={"ID":"b2b9a862-b4dd-424a-a291-94fca0452311","Type":"ContainerDied","Data":"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee"} Sep 20 02:13:06.009741 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.009658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" event={"ID":"b2b9a862-b4dd-424a-a291-94fca0452311","Type":"ContainerDied","Data":"be37aeab941813f1789d4904891db27416576d37d4043badf8ae4e79d018e311"} Sep 20 02:13:06.009741 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.009669 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq" Sep 20 02:13:06.009741 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.009687 2578 scope.go:117] "RemoveContainer" containerID="d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee" Sep 20 02:13:06.028131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.028106 2578 scope.go:117] "RemoveContainer" containerID="d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee" Sep 20 02:13:06.028471 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:13:06.028442 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee\": container with ID starting with d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee not found: ID does not exist" containerID="d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee" Sep 20 02:13:06.028575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.028480 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee"} err="failed to get container status \"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee\": rpc error: code = NotFound desc = could not find container \"d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee\": container with ID starting with d4b41be297907f708a133cc3dde63000ffbcd3282faa18392c145f67e34c9cee not found: ID does not exist" Sep 20 02:13:06.040356 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.040327 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:13:06.044809 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:06.044781 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1750e4-openshift-default-65c9c9b5c6-glkmq"] Sep 20 02:13:07.015931 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:07.015875 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" path="/var/lib/kubelet/pods/b2b9a862-b4dd-424a-a291-94fca0452311/volumes" Sep 20 02:13:10.915512 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:10.915474 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:10.916971 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:10.916945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:10.931196 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:10.931163 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:11.075901 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:11.075873 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:13:20.119889 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.119778 2578 generic.go:356] "Generic (PLEG): container finished" podID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerID="efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f" exitCode=0 Sep 20 02:13:20.119889 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.119851 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerDied","Data":"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f"} Sep 20 02:13:20.120334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.120318 2578 scope.go:117] "RemoveContainer" containerID="efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f" Sep 20 02:13:20.121467 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.121440 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:13:20.121523 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.121510 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:13:20.121560 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:20.121546 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:13:21.127972 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:21.127939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerStarted","Data":"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1"} Sep 20 02:13:21.556894 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:21.556853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:13:21.557128 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:21.557013 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:13:21.557128 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:21.557058 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:13:22.555857 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:22.555797 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:13:22.556400 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:22.555884 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:13:23.555807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:23.555767 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:13:23.556233 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:23.556201 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:13:23.923822 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:23.923791 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:13:24.555846 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:24.555796 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:13:24.556085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:24.555874 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:13:25.555425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:25.555384 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:13:25.556071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:25.556039 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:13:26.559429 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:26.559399 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:13:26.560521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:26.560502 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:13:29.228213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:29.228181 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:13:45.018382 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:13:45.018352 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:14:00.038204 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.038148 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" podUID="a96a58d5-6155-41ba-86dc-6d35789d001c" containerName="main" containerID="cri-o://00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20" gracePeriod=45 Sep 20 02:14:00.290153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.290086 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:14:00.370275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.370241 2578 generic.go:356] "Generic (PLEG): container finished" podID="a96a58d5-6155-41ba-86dc-6d35789d001c" containerID="00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20" exitCode=0 Sep 20 02:14:00.370430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.370287 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" Sep 20 02:14:00.370430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.370317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" event={"ID":"a96a58d5-6155-41ba-86dc-6d35789d001c","Type":"ContainerDied","Data":"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20"} Sep 20 02:14:00.370430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.370360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w" event={"ID":"a96a58d5-6155-41ba-86dc-6d35789d001c","Type":"ContainerDied","Data":"bb8877e253902789911c5fe8a7bb457886689f85bc69d8c34a05f688ebccf7ea"} Sep 20 02:14:00.370430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.370376 2578 scope.go:117] "RemoveContainer" containerID="00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20" Sep 20 02:14:00.379471 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.379442 2578 scope.go:117] "RemoveContainer" containerID="00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20" Sep 20 02:14:00.379737 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:14:00.379718 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20\": container with ID starting with 00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20 not found: ID does not exist" containerID="00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20" Sep 20 02:14:00.379789 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.379747 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20"} err="failed to get container status \"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20\": rpc error: code = NotFound desc = could not find container \"00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20\": container with ID starting with 00016efff71e37e42064c681a7d65b31018df9bb0c1aec1e623a4ca8d7647f20 not found: ID does not exist" Sep 20 02:14:00.410119 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410096 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410184 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410222 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410222 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410208 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv4tf\" (UniqueName: \"kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410300 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410389 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir\") pod \"a96a58d5-6155-41ba-86dc-6d35789d001c\" (UID: \"a96a58d5-6155-41ba-86dc-6d35789d001c\") " Sep 20 02:14:00.410526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410461 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:00.410526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410485 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home" (OuterVolumeSpecName: "home") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:00.410979 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410597 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:00.410979 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410744 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.410979 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410758 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.410979 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.410772 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.412604 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.412577 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm" (OuterVolumeSpecName: "dshm") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:00.412729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.412667 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:14:00.412822 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.412800 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf" (OuterVolumeSpecName: "kube-api-access-nv4tf") pod "a96a58d5-6155-41ba-86dc-6d35789d001c" (UID: "a96a58d5-6155-41ba-86dc-6d35789d001c"). InnerVolumeSpecName "kube-api-access-nv4tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:14:00.511690 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.511661 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4tf\" (UniqueName: \"kubernetes.io/projected/a96a58d5-6155-41ba-86dc-6d35789d001c-kube-api-access-nv4tf\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.511690 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.511685 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96a58d5-6155-41ba-86dc-6d35789d001c-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.511690 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.511695 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96a58d5-6155-41ba-86dc-6d35789d001c-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:00.692868 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.692833 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:14:00.695433 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:00.695412 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-703998/test-model-703998-kserve-b6794ccbf-6qq7w"] Sep 20 02:14:01.016578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:01.016500 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96a58d5-6155-41ba-86dc-6d35789d001c" path="/var/lib/kubelet/pods/a96a58d5-6155-41ba-86dc-6d35789d001c/volumes" Sep 20 02:14:03.425833 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:03.421621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:14:21.992733 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:21.992697 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:14:21.993251 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:21.992982 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" containerID="cri-o://6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86" gracePeriod=30 Sep 20 02:14:22.876759 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.876728 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:14:22.877464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877438 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" containerName="istio-proxy" Sep 20 02:14:22.877464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877465 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" containerName="istio-proxy" Sep 20 02:14:22.877637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877481 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96a58d5-6155-41ba-86dc-6d35789d001c" containerName="main" Sep 20 02:14:22.877637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877490 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96a58d5-6155-41ba-86dc-6d35789d001c" containerName="main" Sep 20 02:14:22.877637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877596 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b9a862-b4dd-424a-a291-94fca0452311" containerName="istio-proxy" Sep 20 02:14:22.877637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.877607 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96a58d5-6155-41ba-86dc-6d35789d001c" containerName="main" Sep 20 02:14:22.880880 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.880864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.883021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.883000 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ed9f38-openshift-default-dockercfg-566c2\"" Sep 20 02:14:22.892755 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.892733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:14:22.915129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915273 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915399 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915399 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915399 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915536 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915536 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztzl\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:22.915536 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:22.915515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016165 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztzl\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.016582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.017058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.017058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.017058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.016861 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.017199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.017123 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.017341 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.017318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.019064 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.019038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.019451 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.019434 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.023651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.023629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.024361 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.024342 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztzl\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl\") pod \"e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.195222 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.195194 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:23.471303 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.471267 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:14:23.472514 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:14:23.472492 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39924e1e_ff2a_48c6_85d3_82334ce668dc.slice/crio-9d5452a7f160bafcab37b1653af83872054c427d37a15240dc7bc54fee4a0115 WatchSource:0}: Error finding container 9d5452a7f160bafcab37b1653af83872054c427d37a15240dc7bc54fee4a0115: Status 404 returned error can't find the container with id 9d5452a7f160bafcab37b1653af83872054c427d37a15240dc7bc54fee4a0115 Sep 20 02:14:23.474869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.474835 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:23.474987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.474930 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:23.474987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.474974 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:23.510537 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:23.510507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerStarted","Data":"9d5452a7f160bafcab37b1653af83872054c427d37a15240dc7bc54fee4a0115"} Sep 20 02:14:24.517209 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:24.517167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerStarted","Data":"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de"} Sep 20 02:14:24.536873 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:24.536830 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podStartSLOduration=2.536815947 podStartE2EDuration="2.536815947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:14:24.535365162 +0000 UTC m=+1316.053680922" watchObservedRunningTime="2026-09-20 02:14:24.536815947 +0000 UTC m=+1316.055131704" Sep 20 02:14:25.195707 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:25.195675 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:25.200584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:25.200561 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:25.523648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:25.523568 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:25.524626 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:25.524609 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:26.152032 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:26.151993 2578 patch_prober.go:28] interesting pod/e2e-inf-b-703998-openshift-default-78669cb68d-f479k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 20 02:14:26.152203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:26.152052 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 20 02:14:27.247255 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.247229 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:14:27.360244 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360158 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360244 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360207 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360244 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360231 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360249 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360270 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360359 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwjrq\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360410 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360463 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs\") pod \"59b6404c-4d47-42ab-9391-b665986090c5\" (UID: \"59b6404c-4d47-42ab-9391-b665986090c5\") " Sep 20 02:14:27.360763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360502 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:27.360763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360592 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:27.360763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360662 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:14:27.361023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360811 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:27.361023 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.360961 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:27.361098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.361024 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.361098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.361044 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.361098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.361063 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b6404c-4d47-42ab-9391-b665986090c5-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.361098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.361080 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.362578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.362557 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:14:27.362825 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.362801 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq" (OuterVolumeSpecName: "kube-api-access-zwjrq") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "kube-api-access-zwjrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:14:27.363088 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.363071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:14:27.363175 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.363136 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59b6404c-4d47-42ab-9391-b665986090c5" (UID: "59b6404c-4d47-42ab-9391-b665986090c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:14:27.462583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.462535 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjrq\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-kube-api-access-zwjrq\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.462583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.462577 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.462583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.462588 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b6404c-4d47-42ab-9391-b665986090c5-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.462583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.462595 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b6404c-4d47-42ab-9391-b665986090c5-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.462583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.462604 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b6404c-4d47-42ab-9391-b665986090c5-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:14:27.538854 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.538820 2578 generic.go:356] "Generic (PLEG): container finished" podID="59b6404c-4d47-42ab-9391-b665986090c5" containerID="6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86" exitCode=0 Sep 20 02:14:27.539067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.538879 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" Sep 20 02:14:27.539067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.538929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerDied","Data":"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86"} Sep 20 02:14:27.539067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.538997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k" event={"ID":"59b6404c-4d47-42ab-9391-b665986090c5","Type":"ContainerDied","Data":"f8c6528986828f29e16c4eeb57cac3a81b64736a8ff5f316c69a08c359556f7c"} Sep 20 02:14:27.539067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.539032 2578 scope.go:117] "RemoveContainer" containerID="6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86" Sep 20 02:14:27.550284 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.550268 2578 scope.go:117] "RemoveContainer" containerID="681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827" Sep 20 02:14:27.559318 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.559242 2578 scope.go:117] "RemoveContainer" containerID="6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86" Sep 20 02:14:27.563524 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:14:27.563105 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86\": container with ID starting with 6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86 not found: ID does not exist" containerID="6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86" Sep 20 02:14:27.563524 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.563144 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86"} err="failed to get container status \"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86\": rpc error: code = NotFound desc = could not find container \"6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86\": container with ID starting with 6dfeffd124dd1bc17735b8f3acc69d4887f047a4f399fa7d5f579a1d26403e86 not found: ID does not exist" Sep 20 02:14:27.563524 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.563169 2578 scope.go:117] "RemoveContainer" containerID="681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827" Sep 20 02:14:27.564157 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:14:27.564126 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827\": container with ID starting with 681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827 not found: ID does not exist" containerID="681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827" Sep 20 02:14:27.564235 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.564155 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827"} err="failed to get container status \"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827\": rpc error: code = NotFound desc = could not find container \"681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827\": container with ID starting with 681a1bf394b467c039169e0bb9bf6cccaf5a11b572a0a3e8cb13affff102c827 not found: ID does not exist" Sep 20 02:14:27.565954 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.565933 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:14:27.571405 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:27.571382 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-703998-openshift-default-78669cb68d-f479k"] Sep 20 02:14:29.015661 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:29.015627 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b6404c-4d47-42ab-9391-b665986090c5" path="/var/lib/kubelet/pods/59b6404c-4d47-42ab-9391-b665986090c5/volumes" Sep 20 02:14:30.718442 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.718403 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:14:30.718958 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.718825 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:30.718958 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.718837 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:30.719096 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.718973 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:30.719096 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.718993 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:30.722649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.722620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.724781 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.724760 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ed9f38-serving-cert\"" Sep 20 02:14:30.736146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.736119 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:14:30.792667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.792626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4x57\" (UniqueName: \"kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.792824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.792719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.792824 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.792786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.792932 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.792849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.894201 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.894161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.894457 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.894230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.894457 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.894293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4x57\" (UniqueName: \"kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.894574 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.894490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.894984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.894967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.896869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.896846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.897048 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.897030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:30.901938 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:30.901894 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4x57\" (UniqueName: \"kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57\") pod \"maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:31.035151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:31.035065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:31.313518 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:31.313489 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:14:31.314576 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:14:31.314546 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e68632_9961_466c_a132_582d8a3d1519.slice/crio-722f3440c87db3de692cd57d6630a680e7c67717950e83e3546b29ce367224d9 WatchSource:0}: Error finding container 722f3440c87db3de692cd57d6630a680e7c67717950e83e3546b29ce367224d9: Status 404 returned error can't find the container with id 722f3440c87db3de692cd57d6630a680e7c67717950e83e3546b29ce367224d9 Sep 20 02:14:31.316401 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:31.316384 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 02:14:31.564452 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:31.564366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" event={"ID":"16e68632-9961-466c-a132-582d8a3d1519","Type":"ContainerStarted","Data":"722f3440c87db3de692cd57d6630a680e7c67717950e83e3546b29ce367224d9"} Sep 20 02:14:32.572373 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:32.572342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" event={"ID":"16e68632-9961-466c-a132-582d8a3d1519","Type":"ContainerStarted","Data":"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10"} Sep 20 02:14:32.572781 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:32.572434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:32.592625 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:32.592581 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" podStartSLOduration=2.042245115 podStartE2EDuration="2.592567393s" totalImagesPullingTime="550.322278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:14:32.590873435 +0000 UTC m=+1324.109189271" watchObservedRunningTime="2026-09-20 02:14:32.592567393 +0000 UTC m=+1324.110883148" Sep 20 02:14:38.583842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:38.583815 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:14:44.420649 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:44.420611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:14:55.525388 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.525352 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:14:55.525787 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.525420 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:14:55.719081 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.719049 2578 generic.go:356] "Generic (PLEG): container finished" podID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerID="5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de" exitCode=0 Sep 20 02:14:55.719251 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.719129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerDied","Data":"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de"} Sep 20 02:14:55.719596 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.719580 2578 scope.go:117] "RemoveContainer" containerID="5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de" Sep 20 02:14:55.720864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.720836 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:55.720951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.720919 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:55.720994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:55.720953 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:56.726609 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:56.726571 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerStarted","Data":"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a"} Sep 20 02:14:57.169160 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.169117 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:14:57.169994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.169973 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:57.170131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.170121 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b6404c-4d47-42ab-9391-b665986090c5" containerName="istio-proxy" Sep 20 02:14:57.173113 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.173090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:14:57.173268 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.173247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.176154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.176128 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-317780-openshift-default-dockercfg-96fcw\"" Sep 20 02:14:57.195781 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.195753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:14:57.195965 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.195942 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:14:57.196049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.195999 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:14:57.264799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.264759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.264810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.264878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.264956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265049 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.264983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.265073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.265118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.265153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.265290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.265193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcmd\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.365717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.365945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.365945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.365945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcmd\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.365945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.365990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366233 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366535 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366586 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366534 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.366742 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.366725 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.368191 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.368173 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.368717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.368675 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.373757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.373736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.373998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.373979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcmd\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd\") pod \"e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.486189 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.486099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:57.797557 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.797531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:14:57.804080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.804040 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:57.804180 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.804114 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:57.804180 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:57.804143 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:14:58.196458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:58.196424 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:14:58.196661 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:58.196482 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:14:58.744744 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:58.744700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerStarted","Data":"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35"} Sep 20 02:14:58.744973 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:58.744772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerStarted","Data":"81a153821fbe7d4ebaec36ca7da51d146d48ca935b8054acad41d448e88770bd"} Sep 20 02:14:58.768785 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:58.768727 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podStartSLOduration=1.7687115420000001 podStartE2EDuration="1.768711542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:14:58.767345007 +0000 UTC m=+1350.285660791" watchObservedRunningTime="2026-09-20 02:14:58.768711542 +0000 UTC m=+1350.287027296" Sep 20 02:14:59.195701 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.195665 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:14:59.196105 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.195721 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:14:59.486340 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.486250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:59.491374 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.491348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:59.754412 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.754328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:14:59.760309 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:14:59.760286 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:15:00.139821 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.139790 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2"] Sep 20 02:15:00.143662 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.143637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" Sep 20 02:15:00.144314 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.144288 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks"] Sep 20 02:15:00.146137 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.146108 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5gscp\"" Sep 20 02:15:00.164098 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.164064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2"] Sep 20 02:15:00.164231 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.164191 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" Sep 20 02:15:00.169400 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.169328 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks"] Sep 20 02:15:00.177237 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.177039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" Sep 20 02:15:00.179336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.179235 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns"] Sep 20 02:15:00.183088 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.182643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" Sep 20 02:15:00.186068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.186012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns"] Sep 20 02:15:00.195717 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.195688 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:15:00.196108 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.195742 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:15:00.223789 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.223660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" Sep 20 02:15:00.229842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.229826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" Sep 20 02:15:00.463071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.463046 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2"] Sep 20 02:15:00.464584 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:15:00.464554 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf700bf03_9f55_427e_9ebf_2f47ab371701.slice/crio-f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566 WatchSource:0}: Error finding container f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566: Status 404 returned error can't find the container with id f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566 Sep 20 02:15:00.560931 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.555226 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks"] Sep 20 02:15:00.561437 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.561368 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns"] Sep 20 02:15:00.563529 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:15:00.563494 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f783da_eba6_4a3c_a708_b1c6ce641200.slice/crio-82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4 WatchSource:0}: Error finding container 82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4: Status 404 returned error can't find the container with id 82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4 Sep 20 02:15:00.762967 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.762840 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" event={"ID":"11f783da-eba6-4a3c-a708-b1c6ce641200","Type":"ContainerStarted","Data":"82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4"} Sep 20 02:15:00.764262 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.764221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" event={"ID":"123c3e2b-d170-4e62-b323-a2625009aa15","Type":"ContainerStarted","Data":"38d3db311939110a26d341db0f0061486062f2d28cb7e7f9079073bea51086a7"} Sep 20 02:15:00.765478 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:00.765454 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" event={"ID":"f700bf03-9f55-427e-9ebf-2f47ab371701","Type":"ContainerStarted","Data":"f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566"} Sep 20 02:15:01.196746 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:01.196718 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 20 02:15:01.197164 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:01.196767 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 20 02:15:02.202028 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:02.201994 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:15:02.202599 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:02.202417 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:15:02.203425 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:02.203391 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:15:03.788951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.788845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" event={"ID":"11f783da-eba6-4a3c-a708-b1c6ce641200","Type":"ContainerStarted","Data":"5acf2f0c5585db7d0a17136ba02aea758c302d507decb9545c650d5fedaad74a"} Sep 20 02:15:03.790539 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.790507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" event={"ID":"123c3e2b-d170-4e62-b323-a2625009aa15","Type":"ContainerStarted","Data":"5dfd55dca729a8dcd0a97997dc3dfcb807d280edfca29d8a9a3e5d8eecabe830"} Sep 20 02:15:03.792029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.792009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" event={"ID":"f700bf03-9f55-427e-9ebf-2f47ab371701","Type":"ContainerStarted","Data":"16b62b85afbf52247c3f7976298fe94575a39ce6bfd907b732790feb6e69b697"} Sep 20 02:15:03.809151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.809102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" podStartSLOduration=1.197383663 podStartE2EDuration="3.809085291s" totalImagesPullingTime="2.611701628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:15:03.808073571 +0000 UTC m=+1355.326389330" watchObservedRunningTime="2026-09-20 02:15:03.809085291 +0000 UTC m=+1355.327401049" Sep 20 02:15:03.823450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.823395 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" podStartSLOduration=1.116738925 podStartE2EDuration="3.823381212s" totalImagesPullingTime="2.706642287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:15:03.821124248 +0000 UTC m=+1355.339440007" watchObservedRunningTime="2026-09-20 02:15:03.823381212 +0000 UTC m=+1355.341696997" Sep 20 02:15:03.834372 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:03.834329 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" podStartSLOduration=1.210016667 podStartE2EDuration="3.834316384s" totalImagesPullingTime="2.624299717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:15:03.833496233 +0000 UTC m=+1355.351811993" watchObservedRunningTime="2026-09-20 02:15:03.834316384 +0000 UTC m=+1355.352632141" Sep 20 02:15:04.800757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.800724 2578 generic.go:356] "Generic (PLEG): container finished" podID="11f783da-eba6-4a3c-a708-b1c6ce641200" containerID="5acf2f0c5585db7d0a17136ba02aea758c302d507decb9545c650d5fedaad74a" exitCode=0 Sep 20 02:15:04.801251 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.800811 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" event={"ID":"11f783da-eba6-4a3c-a708-b1c6ce641200","Type":"ContainerDied","Data":"5acf2f0c5585db7d0a17136ba02aea758c302d507decb9545c650d5fedaad74a"} Sep 20 02:15:04.802447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.802425 2578 generic.go:356] "Generic (PLEG): container finished" podID="123c3e2b-d170-4e62-b323-a2625009aa15" containerID="5dfd55dca729a8dcd0a97997dc3dfcb807d280edfca29d8a9a3e5d8eecabe830" exitCode=0 Sep 20 02:15:04.802556 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.802503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" event={"ID":"123c3e2b-d170-4e62-b323-a2625009aa15","Type":"ContainerDied","Data":"5dfd55dca729a8dcd0a97997dc3dfcb807d280edfca29d8a9a3e5d8eecabe830"} Sep 20 02:15:04.804073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.804044 2578 generic.go:356] "Generic (PLEG): container finished" podID="f700bf03-9f55-427e-9ebf-2f47ab371701" containerID="16b62b85afbf52247c3f7976298fe94575a39ce6bfd907b732790feb6e69b697" exitCode=0 Sep 20 02:15:04.804164 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:04.804119 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" event={"ID":"f700bf03-9f55-427e-9ebf-2f47ab371701","Type":"ContainerDied","Data":"16b62b85afbf52247c3f7976298fe94575a39ce6bfd907b732790feb6e69b697"} Sep 20 02:15:05.974623 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:05.974594 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" Sep 20 02:15:06.026279 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.026255 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" Sep 20 02:15:06.029316 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.029298 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" Sep 20 02:15:06.825722 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.825691 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" Sep 20 02:15:06.825948 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.825692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns" event={"ID":"123c3e2b-d170-4e62-b323-a2625009aa15","Type":"ContainerDied","Data":"38d3db311939110a26d341db0f0061486062f2d28cb7e7f9079073bea51086a7"} Sep 20 02:15:06.825948 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.825843 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d3db311939110a26d341db0f0061486062f2d28cb7e7f9079073bea51086a7" Sep 20 02:15:06.827370 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.827339 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" Sep 20 02:15:06.827485 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.827367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2" event={"ID":"f700bf03-9f55-427e-9ebf-2f47ab371701","Type":"ContainerDied","Data":"f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566"} Sep 20 02:15:06.827485 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.827415 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d5c1acf5a5c78550eb801f9e43563b62b3779b252622f556ba6ffb00551566" Sep 20 02:15:06.828822 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.828800 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" event={"ID":"11f783da-eba6-4a3c-a708-b1c6ce641200","Type":"ContainerDied","Data":"82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4"} Sep 20 02:15:06.828941 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.828838 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82325c14e36001a06d3e7e1b0fa7406cfc3941583e9b9373cfa1063d5cf2e2c4" Sep 20 02:15:06.828941 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:06.828850 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks" Sep 20 02:15:07.614014 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:07.613977 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:15:22.096926 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.096876 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:15:22.097313 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097289 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f783da-eba6-4a3c-a708-b1c6ce641200" containerName="cleanup" Sep 20 02:15:22.097313 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097299 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f783da-eba6-4a3c-a708-b1c6ce641200" containerName="cleanup" Sep 20 02:15:22.097389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097320 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f700bf03-9f55-427e-9ebf-2f47ab371701" containerName="cleanup" Sep 20 02:15:22.097389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097326 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f700bf03-9f55-427e-9ebf-2f47ab371701" containerName="cleanup" Sep 20 02:15:22.097389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097336 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123c3e2b-d170-4e62-b323-a2625009aa15" containerName="cleanup" Sep 20 02:15:22.097389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097342 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="123c3e2b-d170-4e62-b323-a2625009aa15" containerName="cleanup" Sep 20 02:15:22.097513 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097414 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="123c3e2b-d170-4e62-b323-a2625009aa15" containerName="cleanup" Sep 20 02:15:22.097513 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097424 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f783da-eba6-4a3c-a708-b1c6ce641200" containerName="cleanup" Sep 20 02:15:22.097513 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.097433 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f700bf03-9f55-427e-9ebf-2f47ab371701" containerName="cleanup" Sep 20 02:15:22.100501 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.100486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.103364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.103341 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f5813346\"/\"e2e-lifecycle-model-f5813346-kserve-self-signed-certs\"" Sep 20 02:15:22.103505 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.103378 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f5813346\"/\"kube-root-ca.crt\"" Sep 20 02:15:22.103505 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.103344 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f5813346\"/\"openshift-service-ca.crt\"" Sep 20 02:15:22.103505 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.103388 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f5813346\"/\"default-dockercfg-z6np6\"" Sep 20 02:15:22.111430 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.111403 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:15:22.219328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.219328 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.219521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.219521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrrf\" (UniqueName: \"kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.219521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.219521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.219496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrrf\" (UniqueName: \"kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.320970 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.321036 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.320980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.321082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.321019 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.323118 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.323095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.323325 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.323303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.328080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.328059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrrf\" (UniqueName: \"kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf\") pod \"e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.417120 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.417080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:22.716076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.716048 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:15:22.717145 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:15:22.717118 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3995c7cf_b9f1_4405_9bd0_0e59c968e1b6.slice/crio-a242bb059711171a211bcf3ff4c092e28819265694e893482a75a490eff8535c WatchSource:0}: Error finding container a242bb059711171a211bcf3ff4c092e28819265694e893482a75a490eff8535c: Status 404 returned error can't find the container with id a242bb059711171a211bcf3ff4c092e28819265694e893482a75a490eff8535c Sep 20 02:15:22.925002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.924966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" event={"ID":"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6","Type":"ContainerStarted","Data":"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be"} Sep 20 02:15:22.925002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.925012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" event={"ID":"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6","Type":"ContainerStarted","Data":"a242bb059711171a211bcf3ff4c092e28819265694e893482a75a490eff8535c"} Sep 20 02:15:22.944435 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:22.944370 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" podStartSLOduration=0.944349198 podStartE2EDuration="944.349198ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:15:22.941723061 +0000 UTC m=+1374.460038819" watchObservedRunningTime="2026-09-20 02:15:22.944349198 +0000 UTC m=+1374.462664962" Sep 20 02:15:28.964455 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.964418 2578 generic.go:356] "Generic (PLEG): container finished" podID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerID="d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35" exitCode=0 Sep 20 02:15:28.964932 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.964471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerDied","Data":"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35"} Sep 20 02:15:28.964932 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.964925 2578 scope.go:117] "RemoveContainer" containerID="d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35" Sep 20 02:15:28.966124 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.966091 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:15:28.966226 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.966175 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:15:28.966291 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:28.966225 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:15:29.514521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:29.514487 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:15:29.754687 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:29.754644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:15:29.971999 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:29.971966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerStarted","Data":"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1"} Sep 20 02:15:30.486833 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:30.486800 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:15:30.488159 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:30.488133 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 20 02:15:30.488276 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:30.488185 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 20 02:15:31.487305 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:31.487270 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 20 02:15:31.487753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:31.487327 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 20 02:15:32.417630 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:32.417585 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:32.417993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:32.417664 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:32.431146 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:32.431120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:32.486994 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:32.486959 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 20 02:15:32.487167 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:32.487017 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 20 02:15:33.004400 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:33.004368 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:15:33.486736 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:33.486693 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 20 02:15:33.486972 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:33.486760 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 20 02:15:34.487000 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:34.486964 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 20 02:15:34.487488 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:34.487030 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 20 02:15:35.491195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:35.491159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:15:35.491677 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:35.491662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:15:35.492422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:15:35.492401 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:16:01.511938 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:01.511884 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:16:26.322109 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:26.322030 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:16:36.394410 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:36.394371 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2"] Sep 20 02:16:36.396977 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:36.396956 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f5813346-29831175-qgzz2"] Sep 20 02:16:37.016402 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:37.016365 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f700bf03-9f55-427e-9ebf-2f47ab371701" path="/var/lib/kubelet/pods/f700bf03-9f55-427e-9ebf-2f47ab371701/volumes" Sep 20 02:16:44.797692 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:44.797653 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:16:59.818799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:16:59.818733 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" podUID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" containerName="main" containerID="cri-o://be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be" gracePeriod=45 Sep 20 02:17:00.071585 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.071518 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:17:00.147867 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.147837 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgrrf\" (UniqueName: \"kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.147966 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148068 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148047 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148293 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148120 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache\") pod \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\" (UID: \"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6\") " Sep 20 02:17:00.148360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148268 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home" (OuterVolumeSpecName: "home") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:17:00.148360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148338 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:17:00.148463 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148427 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:17:00.148569 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148537 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.148569 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148553 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.148569 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.148589 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.150338 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.150311 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm" (OuterVolumeSpecName: "dshm") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:17:00.150436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.150376 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf" (OuterVolumeSpecName: "kube-api-access-pgrrf") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "kube-api-access-pgrrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:17:00.150436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.150413 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" (UID: "3995c7cf-b9f1-4405-9bd0-0e59c968e1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:17:00.249567 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.249530 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.249567 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.249562 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.249567 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.249573 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgrrf\" (UniqueName: \"kubernetes.io/projected/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6-kube-api-access-pgrrf\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:17:00.547740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.547706 2578 generic.go:356] "Generic (PLEG): container finished" podID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" containerID="be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be" exitCode=0 Sep 20 02:17:00.547945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.547757 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" Sep 20 02:17:00.547945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.547797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" event={"ID":"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6","Type":"ContainerDied","Data":"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be"} Sep 20 02:17:00.547945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.547841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8" event={"ID":"3995c7cf-b9f1-4405-9bd0-0e59c968e1b6","Type":"ContainerDied","Data":"a242bb059711171a211bcf3ff4c092e28819265694e893482a75a490eff8535c"} Sep 20 02:17:00.547945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.547858 2578 scope.go:117] "RemoveContainer" containerID="be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be" Sep 20 02:17:00.558424 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.558407 2578 scope.go:117] "RemoveContainer" containerID="be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be" Sep 20 02:17:00.558688 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:17:00.558670 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be\": container with ID starting with be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be not found: ID does not exist" containerID="be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be" Sep 20 02:17:00.558734 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.558696 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be"} err="failed to get container status \"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be\": rpc error: code = NotFound desc = could not find container \"be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be\": container with ID starting with be6891db9058cf50c9ec6ae99e6427ee2f6a2963028d8b76f7b664e1c19116be not found: ID does not exist" Sep 20 02:17:00.570523 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.570498 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:17:00.572965 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:00.572945 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f5813346/e2e-lifecycle-model-f5813346-kserve-766797c679-5dxb8"] Sep 20 02:17:01.016550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:01.016519 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" path="/var/lib/kubelet/pods/3995c7cf-b9f1-4405-9bd0-0e59c968e1b6/volumes" Sep 20 02:17:01.625419 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:01.625381 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:17:05.896851 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.896817 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:17:05.897294 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.897280 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" containerName="main" Sep 20 02:17:05.897337 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.897295 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" containerName="main" Sep 20 02:17:05.897384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.897375 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3995c7cf-b9f1-4405-9bd0-0e59c968e1b6" containerName="main" Sep 20 02:17:05.901426 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.901411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:05.904531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.904508 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-317780\"/\"openshift-service-ca.crt\"" Sep 20 02:17:05.904679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.904542 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-317780\"/\"default-dockercfg-5nxwn\"" Sep 20 02:17:05.904679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.904542 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-317780\"/\"kube-root-ca.crt\"" Sep 20 02:17:05.904679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.904515 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-317780\"/\"auth-test-model-317780-kserve-self-signed-certs\"" Sep 20 02:17:05.911125 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:05.911105 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:17:06.010470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.010665 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.010665 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pt2\" (UniqueName: \"kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.010665 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.010665 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.010834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.010728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111407 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111565 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111565 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111565 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pt2\" (UniqueName: \"kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111565 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111779 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111997 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.111972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.111997 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.112002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.112156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.112022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.113775 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.113744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.114088 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.114071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.119966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.119939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pt2\" (UniqueName: \"kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2\") pod \"auth-test-model-317780-kserve-595964d76c-s4grn\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.212581 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.212544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:06.496672 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.496646 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:17:06.499163 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:17:06.499138 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode222e765_bc76_47d8_a442_129eea19ce6e.slice/crio-075170aa54a62d97c72169d049d65c24c8704111a5d4df551caf32e36c349a0e WatchSource:0}: Error finding container 075170aa54a62d97c72169d049d65c24c8704111a5d4df551caf32e36c349a0e: Status 404 returned error can't find the container with id 075170aa54a62d97c72169d049d65c24c8704111a5d4df551caf32e36c349a0e Sep 20 02:17:06.592575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.592542 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" event={"ID":"e222e765-bc76-47d8-a442-129eea19ce6e","Type":"ContainerStarted","Data":"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d"} Sep 20 02:17:06.592715 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.592587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" event={"ID":"e222e765-bc76-47d8-a442-129eea19ce6e","Type":"ContainerStarted","Data":"075170aa54a62d97c72169d049d65c24c8704111a5d4df551caf32e36c349a0e"} Sep 20 02:17:06.610266 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:06.610220 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" podStartSLOduration=1.610204813 podStartE2EDuration="1.610204813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:17:06.607429475 +0000 UTC m=+1478.125745241" watchObservedRunningTime="2026-09-20 02:17:06.610204813 +0000 UTC m=+1478.128520570" Sep 20 02:17:16.213711 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:16.213674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:16.213711 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:16.213716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:16.226583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:16.226557 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:16.665715 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:16.665687 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:17:29.120531 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:29.120497 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:17:29.147282 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:29.147252 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:17:31.713021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:17:31.712986 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:18:05.615931 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:18:05.615835 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:18:56.425297 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:18:56.425255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:19:11.333042 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.332999 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:19:11.337554 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.337532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.337706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.337631 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:19:11.339853 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.339830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1246-openshift-default-dockercfg-jxtz5\"" Sep 20 02:19:11.358131 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358256 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzwx\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358256 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358446 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358446 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358527 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.358527 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.358489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzwx\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459685 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459685 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459685 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.459883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.459873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.460230 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.460143 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.460483 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.460420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.460579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.460562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.461000 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.460970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.462350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.462322 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.462542 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.462526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.473716 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.473689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzwx\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.476476 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.476454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token\") pod \"e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.651445 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.651410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:11.931663 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.929560 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:19:11.932487 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:19:11.932455 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3f3829_13af_4e8a_b5fa_09cfc997ed36.slice/crio-a3da73da2961789194e392387c2ac0e64450e15b0a6c55763085349883f691fc WatchSource:0}: Error finding container a3da73da2961789194e392387c2ac0e64450e15b0a6c55763085349883f691fc: Status 404 returned error can't find the container with id a3da73da2961789194e392387c2ac0e64450e15b0a6c55763085349883f691fc Sep 20 02:19:11.934723 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.934680 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:19:11.934821 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.934773 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:19:11.934866 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:11.934816 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:19:12.376991 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:12.376893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerStarted","Data":"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d"} Sep 20 02:19:12.376991 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:12.376961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerStarted","Data":"a3da73da2961789194e392387c2ac0e64450e15b0a6c55763085349883f691fc"} Sep 20 02:19:12.395038 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:12.394975 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podStartSLOduration=1.394960424 podStartE2EDuration="1.394960424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:19:12.39479482 +0000 UTC m=+1603.913110579" watchObservedRunningTime="2026-09-20 02:19:12.394960424 +0000 UTC m=+1603.913276182" Sep 20 02:19:12.652557 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:12.652523 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:12.658199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:12.658174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:13.383144 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:13.383111 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:13.383964 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:13.383946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:19:20.712166 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.712081 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:19:20.715779 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.715763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.718346 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.718324 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dd1246-dockercfg-jfgsz\"" Sep 20 02:19:20.718475 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.718375 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dd1246\"" Sep 20 02:19:20.727314 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.727277 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:19:20.757883 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.757850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bxsw\" (UniqueName: \"kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.758034 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.757995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.859297 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.859234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bxsw\" (UniqueName: \"kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.859492 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.859406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.860175 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.860151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:20.866820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:20.866799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bxsw\" (UniqueName: \"kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw\") pod \"payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:21.029254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:21.029180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:21.308064 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:21.308035 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:19:21.309053 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:19:21.309030 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064480f3_b97f_454d_a357_5578b334ba04.slice/crio-70799bebb745de3e88776734dd3eeacf5d913750078f2fd09171d41bbfe8fdbd WatchSource:0}: Error finding container 70799bebb745de3e88776734dd3eeacf5d913750078f2fd09171d41bbfe8fdbd: Status 404 returned error can't find the container with id 70799bebb745de3e88776734dd3eeacf5d913750078f2fd09171d41bbfe8fdbd Sep 20 02:19:21.436298 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:21.436265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" event={"ID":"064480f3-b97f-454d-a357-5578b334ba04","Type":"ContainerStarted","Data":"70799bebb745de3e88776734dd3eeacf5d913750078f2fd09171d41bbfe8fdbd"} Sep 20 02:19:23.450605 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:23.450559 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" event={"ID":"064480f3-b97f-454d-a357-5578b334ba04","Type":"ContainerStarted","Data":"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2"} Sep 20 02:19:23.451101 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:23.450630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:23.468354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:23.468264 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" podStartSLOduration=1.567549719 podStartE2EDuration="3.468247798s" totalImagesPullingTime="1.900698079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:19:23.465046851 +0000 UTC m=+1614.983362610" watchObservedRunningTime="2026-09-20 02:19:23.468247798 +0000 UTC m=+1614.986563572" Sep 20 02:19:34.459568 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:34.459535 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:19:37.555322 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:37.555283 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:19:37.576129 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:37.576102 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:19:37.582071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:37.582041 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:19:52.577344 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.577293 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" podUID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" containerName="main" containerID="cri-o://e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a" gracePeriod=45 Sep 20 02:19:52.603525 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.603379 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" podUID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" containerName="main" containerID="cri-o://eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea" gracePeriod=45 Sep 20 02:19:52.604329 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.604297 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" podUID="a49db703-24c3-4212-9656-bd0405ecf3ac" containerName="main" containerID="cri-o://9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace" gracePeriod=45 Sep 20 02:19:52.876322 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.876295 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:19:52.916676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.916654 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:19:52.920393 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:52.920364 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:19:53.005545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005585 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005630 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.005721 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005690 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005737 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005766 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7clh\" (UniqueName: \"kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005792 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005887 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005952 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home\") pod \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\" (UID: \"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005988 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.006070 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006039 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.006540 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006135 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd4b4\" (UniqueName: \"kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4\") pod \"a49db703-24c3-4212-9656-bd0405ecf3ac\" (UID: \"a49db703-24c3-4212-9656-bd0405ecf3ac\") " Sep 20 02:19:53.006540 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006181 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkqkg\" (UniqueName: \"kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg\") pod \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\" (UID: \"f3c747e1-1d4d-4184-b5e4-b0206073eff1\") " Sep 20 02:19:53.008474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005850 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home" (OuterVolumeSpecName: "home") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.005866 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006081 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006164 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.006368 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home" (OuterVolumeSpecName: "home") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008724 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008770 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home" (OuterVolumeSpecName: "home") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.008839 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008792 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm" (OuterVolumeSpecName: "dshm") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.009092 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008871 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:19:53.009092 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.008985 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm" (OuterVolumeSpecName: "dshm") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.009092 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.009002 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.009259 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.009157 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:19:53.009441 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.009416 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg" (OuterVolumeSpecName: "kube-api-access-tkqkg") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "kube-api-access-tkqkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:19:53.009594 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.009557 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:19:53.010283 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.010261 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh" (OuterVolumeSpecName: "kube-api-access-c7clh") pod "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" (UID: "ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e"). InnerVolumeSpecName "kube-api-access-c7clh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:19:53.011142 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.011113 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm" (OuterVolumeSpecName: "dshm") pod "f3c747e1-1d4d-4184-b5e4-b0206073eff1" (UID: "f3c747e1-1d4d-4184-b5e4-b0206073eff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:19:53.011240 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.011143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4" (OuterVolumeSpecName: "kube-api-access-gd4b4") pod "a49db703-24c3-4212-9656-bd0405ecf3ac" (UID: "a49db703-24c3-4212-9656-bd0405ecf3ac"). InnerVolumeSpecName "kube-api-access-gd4b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107471 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4b4\" (UniqueName: \"kubernetes.io/projected/a49db703-24c3-4212-9656-bd0405ecf3ac-kube-api-access-gd4b4\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107506 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqkg\" (UniqueName: \"kubernetes.io/projected/f3c747e1-1d4d-4184-b5e4-b0206073eff1-kube-api-access-tkqkg\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107521 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107534 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107548 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107560 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107572 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107584 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107596 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107608 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49db703-24c3-4212-9656-bd0405ecf3ac-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107620 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7clh\" (UniqueName: \"kubernetes.io/projected/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-kube-api-access-c7clh\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107636 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107650 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107663 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107676 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107690 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107703 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49db703-24c3-4212-9656-bd0405ecf3ac-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.107840 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.107715 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c747e1-1d4d-4184-b5e4-b0206073eff1-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:19:53.644351 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.644322 2578 generic.go:356] "Generic (PLEG): container finished" podID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" containerID="eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea" exitCode=0 Sep 20 02:19:53.644753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.644359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" event={"ID":"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e","Type":"ContainerDied","Data":"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea"} Sep 20 02:19:53.644753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.644372 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" Sep 20 02:19:53.644753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.644419 2578 scope.go:117] "RemoveContainer" containerID="eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea" Sep 20 02:19:53.644753 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.644409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk" event={"ID":"ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e","Type":"ContainerDied","Data":"882a74c17d58af049c8b6f7113988d1b3d9a19b7d332dc6c984732e32abe6618"} Sep 20 02:19:53.646014 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.645983 2578 generic.go:356] "Generic (PLEG): container finished" podID="a49db703-24c3-4212-9656-bd0405ecf3ac" containerID="9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace" exitCode=0 Sep 20 02:19:53.646109 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.646033 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" Sep 20 02:19:53.646109 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.646059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" event={"ID":"a49db703-24c3-4212-9656-bd0405ecf3ac","Type":"ContainerDied","Data":"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace"} Sep 20 02:19:53.646109 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.646089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf" event={"ID":"a49db703-24c3-4212-9656-bd0405ecf3ac","Type":"ContainerDied","Data":"bf64bb0900943cde926928b5204735d876c1cf8751a24ade999eb65c6cfc6851"} Sep 20 02:19:53.647834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.647809 2578 generic.go:356] "Generic (PLEG): container finished" podID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" containerID="e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a" exitCode=0 Sep 20 02:19:53.647987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.647879 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" Sep 20 02:19:53.647987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.647940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" event={"ID":"f3c747e1-1d4d-4184-b5e4-b0206073eff1","Type":"ContainerDied","Data":"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a"} Sep 20 02:19:53.647987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.647969 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt" event={"ID":"f3c747e1-1d4d-4184-b5e4-b0206073eff1","Type":"ContainerDied","Data":"a40bc272e1da30357ccff75c5b3a91db580c8a2c28e742583d3b56d7d6f6697c"} Sep 20 02:19:53.655993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.655978 2578 scope.go:117] "RemoveContainer" containerID="eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea" Sep 20 02:19:53.656233 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:19:53.656217 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea\": container with ID starting with eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea not found: ID does not exist" containerID="eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea" Sep 20 02:19:53.656288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.656240 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea"} err="failed to get container status \"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea\": rpc error: code = NotFound desc = could not find container \"eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea\": container with ID starting with eceb677be7d86d17effd584bf62f8b849902172c6a57e320cca6317b631c5fea not found: ID does not exist" Sep 20 02:19:53.656288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.656256 2578 scope.go:117] "RemoveContainer" containerID="9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace" Sep 20 02:19:53.664597 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.664576 2578 scope.go:117] "RemoveContainer" containerID="9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace" Sep 20 02:19:53.664895 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:19:53.664873 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace\": container with ID starting with 9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace not found: ID does not exist" containerID="9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace" Sep 20 02:19:53.664975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.664917 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace"} err="failed to get container status \"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace\": rpc error: code = NotFound desc = could not find container \"9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace\": container with ID starting with 9d241cf88cdd29caf28a20500d97400ba0ff7262b07209684034648980b44ace not found: ID does not exist" Sep 20 02:19:53.664975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.664936 2578 scope.go:117] "RemoveContainer" containerID="e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a" Sep 20 02:19:53.674664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.674645 2578 scope.go:117] "RemoveContainer" containerID="e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a" Sep 20 02:19:53.674975 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:19:53.674955 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a\": container with ID starting with e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a not found: ID does not exist" containerID="e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a" Sep 20 02:19:53.675058 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.674980 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a"} err="failed to get container status \"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a\": rpc error: code = NotFound desc = could not find container \"e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a\": container with ID starting with e18eee34a06d592171bd225093a6945138db72bb19ad022833799da063b7797a not found: ID does not exist" Sep 20 02:19:53.691397 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.691373 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:19:53.699373 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.699347 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/simulator-w0-8195ba-kserve-649dd587c-qj7hk"] Sep 20 02:19:53.724612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.724589 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:19:53.733403 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.733381 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/distinct-w0-8195ba-kserve-7b6ff8cc9-qhkdt"] Sep 20 02:19:53.749436 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.749414 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:19:53.753308 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:53.753288 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8195ba/premium-w0-8195ba-kserve-b574b64b8-r9pkf"] Sep 20 02:19:55.016219 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:55.016188 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49db703-24c3-4212-9656-bd0405ecf3ac" path="/var/lib/kubelet/pods/a49db703-24c3-4212-9656-bd0405ecf3ac/volumes" Sep 20 02:19:55.016593 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:55.016560 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" path="/var/lib/kubelet/pods/ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e/volumes" Sep 20 02:19:55.016953 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:19:55.016921 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" path="/var/lib/kubelet/pods/f3c747e1-1d4d-4184-b5e4-b0206073eff1/volumes" Sep 20 02:20:15.325288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:15.325255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:20:26.861450 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.861416 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerID="d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d" exitCode=0 Sep 20 02:20:26.861859 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.861494 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerDied","Data":"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d"} Sep 20 02:20:26.862030 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.862008 2578 scope.go:117] "RemoveContainer" containerID="d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d" Sep 20 02:20:26.863186 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.863159 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:20:26.863260 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.863224 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:20:26.863296 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:26.863257 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 20 02:20:27.394091 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.394041 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:20:27.394359 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.394306 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" podUID="064480f3-b97f-454d-a357-5578b334ba04" containerName="payload-pre-processing" containerID="cri-o://44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2" gracePeriod=30 Sep 20 02:20:27.699080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.699058 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:20:27.763756 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.763710 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bxsw\" (UniqueName: \"kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw\") pod \"064480f3-b97f-454d-a357-5578b334ba04\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " Sep 20 02:20:27.764020 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.763973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume\") pod \"064480f3-b97f-454d-a357-5578b334ba04\" (UID: \"064480f3-b97f-454d-a357-5578b334ba04\") " Sep 20 02:20:27.764398 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.764361 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "064480f3-b97f-454d-a357-5578b334ba04" (UID: "064480f3-b97f-454d-a357-5578b334ba04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:20:27.766221 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.766191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw" (OuterVolumeSpecName: "kube-api-access-5bxsw") pod "064480f3-b97f-454d-a357-5578b334ba04" (UID: "064480f3-b97f-454d-a357-5578b334ba04"). InnerVolumeSpecName "kube-api-access-5bxsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:27.865963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.865921 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bxsw\" (UniqueName: \"kubernetes.io/projected/064480f3-b97f-454d-a357-5578b334ba04-kube-api-access-5bxsw\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:27.865963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.865953 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064480f3-b97f-454d-a357-5578b334ba04-plugins-config-volume\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:27.871983 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.871927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerStarted","Data":"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621"} Sep 20 02:20:27.873941 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.873899 2578 generic.go:356] "Generic (PLEG): container finished" podID="064480f3-b97f-454d-a357-5578b334ba04" containerID="44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2" exitCode=0 Sep 20 02:20:27.874116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.873951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" event={"ID":"064480f3-b97f-454d-a357-5578b334ba04","Type":"ContainerDied","Data":"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2"} Sep 20 02:20:27.874116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.873971 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" Sep 20 02:20:27.874116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.874004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm" event={"ID":"064480f3-b97f-454d-a357-5578b334ba04","Type":"ContainerDied","Data":"70799bebb745de3e88776734dd3eeacf5d913750078f2fd09171d41bbfe8fdbd"} Sep 20 02:20:27.874116 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.874032 2578 scope.go:117] "RemoveContainer" containerID="44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2" Sep 20 02:20:27.885037 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.885018 2578 scope.go:117] "RemoveContainer" containerID="44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2" Sep 20 02:20:27.885303 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:20:27.885287 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2\": container with ID starting with 44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2 not found: ID does not exist" containerID="44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2" Sep 20 02:20:27.885352 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.885309 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2"} err="failed to get container status \"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2\": rpc error: code = NotFound desc = could not find container \"44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2\": container with ID starting with 44499d421bc2462c5af25561c6b1b0d9b5640fac8309e1bd6d7e442645f669e2 not found: ID does not exist" Sep 20 02:20:27.904785 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.904763 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:20:27.908021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:27.908002 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1246-946648898-4hshm"] Sep 20 02:20:28.383149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:28.383119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:28.652290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:28.652251 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:28.653584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:28.653543 2578 patch_prober.go:28] interesting pod/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 20 02:20:28.653724 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:28.653610 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 20 02:20:29.016679 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:29.016604 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064480f3-b97f-454d-a357-5578b334ba04" path="/var/lib/kubelet/pods/064480f3-b97f-454d-a357-5578b334ba04/volumes" Sep 20 02:20:29.653074 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:29.652574 2578 patch_prober.go:28] interesting pod/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 20 02:20:29.653074 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:29.652656 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 20 02:20:30.653039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:30.652995 2578 patch_prober.go:28] interesting pod/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 20 02:20:30.653548 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:30.653076 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 20 02:20:31.652337 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:31.652297 2578 patch_prober.go:28] interesting pod/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 20 02:20:31.652627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:31.652595 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 20 02:20:32.651892 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:32.651845 2578 patch_prober.go:28] interesting pod/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 20 02:20:32.652413 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:32.651944 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 20 02:20:33.656603 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:33.656571 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:33.657582 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:33.657561 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:34.036453 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:34.036370 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:20:34.917986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:34.917943 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" containerID="cri-o://51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621" gracePeriod=30 Sep 20 02:20:40.166852 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.166818 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:40.288864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288771 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.288864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288809 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.288864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288837 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.288864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288856 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288884 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288949 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.288986 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289004 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrzwx\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289104 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert\") pod \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\" (UID: \"9f3f3829-13af-4e8a-b5fa-09cfc997ed36\") " Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289157 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289169 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:40.289215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289254 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289509 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289529 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289542 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289550 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:20:40.289641 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.289555 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.291449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.291427 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:20:40.291543 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.291435 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:40.291543 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.291454 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx" (OuterVolumeSpecName: "kube-api-access-zrzwx") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "kube-api-access-zrzwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:40.291543 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.291517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f3f3829-13af-4e8a-b5fa-09cfc997ed36" (UID: "9f3f3829-13af-4e8a-b5fa-09cfc997ed36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:40.390763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.390726 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.390763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.390757 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.390763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.390768 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.390993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.390780 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrzwx\" (UniqueName: \"kubernetes.io/projected/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-kube-api-access-zrzwx\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.390993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.390792 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f3f3829-13af-4e8a-b5fa-09cfc997ed36-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:40.955809 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.955776 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerID="51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621" exitCode=0 Sep 20 02:20:40.956039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.955827 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" Sep 20 02:20:40.956039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.955875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerDied","Data":"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621"} Sep 20 02:20:40.956039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.955962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw" event={"ID":"9f3f3829-13af-4e8a-b5fa-09cfc997ed36","Type":"ContainerDied","Data":"a3da73da2961789194e392387c2ac0e64450e15b0a6c55763085349883f691fc"} Sep 20 02:20:40.956039 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.955993 2578 scope.go:117] "RemoveContainer" containerID="51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621" Sep 20 02:20:40.966423 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.966409 2578 scope.go:117] "RemoveContainer" containerID="d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d" Sep 20 02:20:40.975018 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.975002 2578 scope.go:117] "RemoveContainer" containerID="51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621" Sep 20 02:20:40.975292 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:20:40.975268 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621\": container with ID starting with 51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621 not found: ID does not exist" containerID="51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621" Sep 20 02:20:40.975434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.975298 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621"} err="failed to get container status \"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621\": rpc error: code = NotFound desc = could not find container \"51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621\": container with ID starting with 51aa3c5afa885f7a2aaddf30f629581717ff2d6e20fbb378b9a65ad74fae0621 not found: ID does not exist" Sep 20 02:20:40.975434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.975315 2578 scope.go:117] "RemoveContainer" containerID="d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d" Sep 20 02:20:40.975743 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:20:40.975714 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d\": container with ID starting with d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d not found: ID does not exist" containerID="d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d" Sep 20 02:20:40.975835 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.975749 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d"} err="failed to get container status \"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d\": rpc error: code = NotFound desc = could not find container \"d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d\": container with ID starting with d2973f924730fc90a990d024882ef1b61713ee070bfdd8619aa1a322adeb437d not found: ID does not exist" Sep 20 02:20:40.979482 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.979458 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:20:40.982265 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:40.982239 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1246-openshift-default-59969c48dd-87qlw"] Sep 20 02:20:41.015730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:41.015703 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" path="/var/lib/kubelet/pods/9f3f3829-13af-4e8a-b5fa-09cfc997ed36/volumes" Sep 20 02:20:44.824289 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:44.824250 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:20:44.824772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:44.824505 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" podUID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" containerName="maas-api" containerID="cri-o://b01dbd07287b305687cf275b1a7e95a0fafd79cb9e2c17315dd66b56527b2dd4" gracePeriod=30 Sep 20 02:20:44.983474 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:44.983431 2578 generic.go:356] "Generic (PLEG): container finished" podID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" containerID="b01dbd07287b305687cf275b1a7e95a0fafd79cb9e2c17315dd66b56527b2dd4" exitCode=0 Sep 20 02:20:44.983652 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:44.983517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" event={"ID":"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3","Type":"ContainerDied","Data":"b01dbd07287b305687cf275b1a7e95a0fafd79cb9e2c17315dd66b56527b2dd4"} Sep 20 02:20:45.069972 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.069946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:20:45.234793 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.234757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fh58\" (UniqueName: \"kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58\") pod \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " Sep 20 02:20:45.234988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.234858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls\") pod \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " Sep 20 02:20:45.234988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.234939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs\") pod \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " Sep 20 02:20:45.234988 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.234972 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca\") pod \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\" (UID: \"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3\") " Sep 20 02:20:45.235506 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.235419 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" (UID: "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:20:45.237194 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.237159 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" (UID: "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:20:45.237303 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.237209 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" (UID: "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:20:45.237303 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.237219 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58" (OuterVolumeSpecName: "kube-api-access-5fh58") pod "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" (UID: "3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3"). InnerVolumeSpecName "kube-api-access-5fh58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:45.336799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.336758 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:45.336799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.336794 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:45.336799 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.336803 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:45.337156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.336812 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fh58\" (UniqueName: \"kubernetes.io/projected/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3-kube-api-access-5fh58\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:45.991102 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.991067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" event={"ID":"3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3","Type":"ContainerDied","Data":"9791572fff88d5eefaff648b7ad12535f5a4b6f68b422f4051648763e95cb84e"} Sep 20 02:20:45.991522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.991107 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf" Sep 20 02:20:45.991522 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:45.991132 2578 scope.go:117] "RemoveContainer" containerID="b01dbd07287b305687cf275b1a7e95a0fafd79cb9e2c17315dd66b56527b2dd4" Sep 20 02:20:46.011937 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:46.011898 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:20:46.015710 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:46.015684 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4559b3-c6c88f597-w26jf"] Sep 20 02:20:47.016077 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:47.016046 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" path="/var/lib/kubelet/pods/3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3/volumes" Sep 20 02:20:51.430940 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:51.430838 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:20:51.431354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:51.431110 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" containerID="cri-o://1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1" gracePeriod=30 Sep 20 02:20:52.020842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:52.020811 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:20:56.561101 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.561064 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 20 02:20:56.561464 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.561135 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 20 02:20:56.685614 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.685587 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:20:56.746963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.746835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.746963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.746890 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.746963 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.746949 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwxkk\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.746995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747011 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747050 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747113 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:56.747254 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747247 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747271 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket\") pod \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\" (UID: \"b476887d-ae63-491c-9d46-4ab4cf2c67cf\") " Sep 20 02:20:56.747578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747409 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:56.747578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747474 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:20:56.747751 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747651 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:56.747858 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.747826 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:56.748153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.748009 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.748248 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.748166 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.748248 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.748186 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.748248 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.748200 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.749503 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.749474 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:20:56.749584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.749536 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:20:56.749584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.749567 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk" (OuterVolumeSpecName: "kube-api-access-jwxkk") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "kube-api-access-jwxkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:56.749584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.749560 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "b476887d-ae63-491c-9d46-4ab4cf2c67cf" (UID: "b476887d-ae63-491c-9d46-4ab4cf2c67cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:20:56.849422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.849376 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.849422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.849415 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.849422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.849424 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxkk\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-kube-api-access-jwxkk\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.849422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.849433 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b476887d-ae63-491c-9d46-4ab4cf2c67cf-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:56.849422 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:56.849441 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b476887d-ae63-491c-9d46-4ab4cf2c67cf-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:20:57.060829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.060794 2578 generic.go:356] "Generic (PLEG): container finished" podID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerID="1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1" exitCode=0 Sep 20 02:20:57.060998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.060845 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" Sep 20 02:20:57.060998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.060873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerDied","Data":"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1"} Sep 20 02:20:57.060998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.060960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx" event={"ID":"b476887d-ae63-491c-9d46-4ab4cf2c67cf","Type":"ContainerDied","Data":"4dc94f6f532eb7d6f62c712d22a2604f32e1aa8ad1af3d6cf899ee400aaa526e"} Sep 20 02:20:57.060998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.060993 2578 scope.go:117] "RemoveContainer" containerID="1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1" Sep 20 02:20:57.070468 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.070450 2578 scope.go:117] "RemoveContainer" containerID="efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f" Sep 20 02:20:57.077713 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.077690 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:20:57.079664 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.079639 2578 scope.go:117] "RemoveContainer" containerID="1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1" Sep 20 02:20:57.080031 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:20:57.080000 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1\": container with ID starting with 1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1 not found: ID does not exist" containerID="1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1" Sep 20 02:20:57.080137 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.080042 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1"} err="failed to get container status \"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1\": rpc error: code = NotFound desc = could not find container \"1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1\": container with ID starting with 1ffbe487c576d6950a068b89c010b1a37719481b3682bccd73c6d917bd1625b1 not found: ID does not exist" Sep 20 02:20:57.080137 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.080065 2578 scope.go:117] "RemoveContainer" containerID="efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f" Sep 20 02:20:57.080503 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:20:57.080418 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f\": container with ID starting with efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f not found: ID does not exist" containerID="efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f" Sep 20 02:20:57.080503 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.080454 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f"} err="failed to get container status \"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f\": rpc error: code = NotFound desc = could not find container \"efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f\": container with ID starting with efc339f1abaf66b3eabfa76d8181a78a4b96a97eacfc3686a5a57cb6cc4da86f not found: ID does not exist" Sep 20 02:20:57.082900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:57.082881 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4559b3-openshift-default-7c5f85f54-w98fx"] Sep 20 02:20:59.016545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:20:59.016514 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" path="/var/lib/kubelet/pods/b476887d-ae63-491c-9d46-4ab4cf2c67cf/volumes" Sep 20 02:21:02.382493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:02.382451 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks"] Sep 20 02:21:02.385848 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:02.385816 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90811b-29831175-864ks"] Sep 20 02:21:03.015998 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:03.015966 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f783da-eba6-4a3c-a708-b1c6ce641200" path="/var/lib/kubelet/pods/11f783da-eba6-4a3c-a708-b1c6ce641200/volumes" Sep 20 02:21:19.812783 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:19.812739 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:21:29.310877 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:29.310837 2578 scope.go:117] "RemoveContainer" containerID="5acf2f0c5585db7d0a17136ba02aea758c302d507decb9545c650d5fedaad74a" Sep 20 02:21:29.319935 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:29.319897 2578 scope.go:117] "RemoveContainer" containerID="16b62b85afbf52247c3f7976298fe94575a39ce6bfd907b732790feb6e69b697" Sep 20 02:21:30.838841 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:30.838796 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:21:30.847527 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:30.847496 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:21:41.413637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:41.413588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:21:41.631861 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:41.631816 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns"] Sep 20 02:21:41.636706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:41.636676 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b15051-29831175-hkdns"] Sep 20 02:21:43.022643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:43.022605 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123c3e2b-d170-4e62-b323-a2625009aa15" path="/var/lib/kubelet/pods/123c3e2b-d170-4e62-b323-a2625009aa15/volumes" Sep 20 02:21:45.859597 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:45.859543 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" podUID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" containerName="main" containerID="cri-o://54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926" gracePeriod=45 Sep 20 02:21:45.869350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:45.869316 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" podUID="cadfd41c-4397-4982-a4e8-7346fac046e6" containerName="main" containerID="cri-o://0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a" gracePeriod=45 Sep 20 02:21:46.135573 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.135550 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:21:46.138490 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.138470 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:21:46.239463 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.239647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239479 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.239647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239534 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.239647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239562 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.239647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkx9v\" (UniqueName: \"kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.239647 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239634 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.239955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239660 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.240029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home" (OuterVolumeSpecName: "home") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.240029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239793 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.240029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239885 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home" (OuterVolumeSpecName: "home") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.240029 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.239941 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.240215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240033 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4zp4\" (UniqueName: \"kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.240215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240081 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache\") pod \"cadfd41c-4397-4982-a4e8-7346fac046e6\" (UID: \"cadfd41c-4397-4982-a4e8-7346fac046e6\") " Sep 20 02:21:46.240215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240115 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.240215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240141 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.240215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240180 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm\") pod \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\" (UID: \"eba8cd0c-6a62-4c78-8506-4aea061aad6a\") " Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240558 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240733 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240754 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240770 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240783 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.240814 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240798 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.241223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.240848 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.242545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.242510 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:21:46.242545 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.242523 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v" (OuterVolumeSpecName: "kube-api-access-nkx9v") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "kube-api-access-nkx9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:21:46.242716 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.242647 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm" (OuterVolumeSpecName: "dshm") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.242986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.242960 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4" (OuterVolumeSpecName: "kube-api-access-g4zp4") pod "cadfd41c-4397-4982-a4e8-7346fac046e6" (UID: "cadfd41c-4397-4982-a4e8-7346fac046e6"). InnerVolumeSpecName "kube-api-access-g4zp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:21:46.243189 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.243166 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm" (OuterVolumeSpecName: "dshm") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:21:46.243253 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.243214 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eba8cd0c-6a62-4c78-8506-4aea061aad6a" (UID: "eba8cd0c-6a62-4c78-8506-4aea061aad6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:21:46.342259 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342228 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zp4\" (UniqueName: \"kubernetes.io/projected/cadfd41c-4397-4982-a4e8-7346fac046e6-kube-api-access-g4zp4\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342259 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342257 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342269 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba8cd0c-6a62-4c78-8506-4aea061aad6a-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342278 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba8cd0c-6a62-4c78-8506-4aea061aad6a-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342286 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cadfd41c-4397-4982-a4e8-7346fac046e6-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342293 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkx9v\" (UniqueName: \"kubernetes.io/projected/eba8cd0c-6a62-4c78-8506-4aea061aad6a-kube-api-access-nkx9v\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.342458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.342301 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cadfd41c-4397-4982-a4e8-7346fac046e6-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:21:46.351299 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.351264 2578 generic.go:356] "Generic (PLEG): container finished" podID="cadfd41c-4397-4982-a4e8-7346fac046e6" containerID="0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a" exitCode=0 Sep 20 02:21:46.351449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.351318 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" Sep 20 02:21:46.351449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.351348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" event={"ID":"cadfd41c-4397-4982-a4e8-7346fac046e6","Type":"ContainerDied","Data":"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a"} Sep 20 02:21:46.351449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.351392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr" event={"ID":"cadfd41c-4397-4982-a4e8-7346fac046e6","Type":"ContainerDied","Data":"7aa86b84b22bd3143f3cee138b4a5d5200ca9ef5ed9c8f3cd0baf80aec4188ac"} Sep 20 02:21:46.351449 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.351416 2578 scope.go:117] "RemoveContainer" containerID="0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a" Sep 20 02:21:46.353173 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.353152 2578 generic.go:356] "Generic (PLEG): container finished" podID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" containerID="54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926" exitCode=0 Sep 20 02:21:46.353290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.353188 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" Sep 20 02:21:46.353290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.353239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" event={"ID":"eba8cd0c-6a62-4c78-8506-4aea061aad6a","Type":"ContainerDied","Data":"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926"} Sep 20 02:21:46.353290 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.353275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh" event={"ID":"eba8cd0c-6a62-4c78-8506-4aea061aad6a","Type":"ContainerDied","Data":"91aa9e6de361475a1b98485864dbc2aea5d00f9a0ff98c704a3ff170864dfe41"} Sep 20 02:21:46.366054 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.366037 2578 scope.go:117] "RemoveContainer" containerID="0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a" Sep 20 02:21:46.366350 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:21:46.366310 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a\": container with ID starting with 0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a not found: ID does not exist" containerID="0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a" Sep 20 02:21:46.366350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.366335 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a"} err="failed to get container status \"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a\": rpc error: code = NotFound desc = could not find container \"0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a\": container with ID starting with 0dd33aef748eb801526c1dd8dc7fa5e93e996305096135bbd550f3fc0e5da09a not found: ID does not exist" Sep 20 02:21:46.366350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.366349 2578 scope.go:117] "RemoveContainer" containerID="54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926" Sep 20 02:21:46.374859 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.374836 2578 scope.go:117] "RemoveContainer" containerID="54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926" Sep 20 02:21:46.375532 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:21:46.375225 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926\": container with ID starting with 54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926 not found: ID does not exist" containerID="54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926" Sep 20 02:21:46.375532 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.375284 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926"} err="failed to get container status \"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926\": rpc error: code = NotFound desc = could not find container \"54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926\": container with ID starting with 54e35eccaf2864deda3577558ea8f1365f713b2de6be4601d22d34fae46db926 not found: ID does not exist" Sep 20 02:21:46.378038 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.378016 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:21:46.381584 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.381563 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/premium-w4-b15051-kserve-7fd996d886-wjlwh"] Sep 20 02:21:46.390670 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.390649 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:21:46.394238 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:46.394217 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b15051/simulator-w4-b15051-kserve-7df9778df9-pdjdr"] Sep 20 02:21:47.016205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:47.016176 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cadfd41c-4397-4982-a4e8-7346fac046e6" path="/var/lib/kubelet/pods/cadfd41c-4397-4982-a4e8-7346fac046e6/volumes" Sep 20 02:21:47.016591 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:47.016542 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" path="/var/lib/kubelet/pods/eba8cd0c-6a62-4c78-8506-4aea061aad6a/volumes" Sep 20 02:21:59.029147 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:21:59.029098 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:22:01.100434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:01.100399 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:22:10.031050 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:10.031006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:22:16.120788 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.120742 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" podUID="e222e765-bc76-47d8-a442-129eea19ce6e" containerName="main" containerID="cri-o://783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d" gracePeriod=45 Sep 20 02:22:16.366969 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.366946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:22:16.428212 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428182 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428220 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2pt2\" (UniqueName: \"kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428281 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428353 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428404 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs\") pod \"e222e765-bc76-47d8-a442-129eea19ce6e\" (UID: \"e222e765-bc76-47d8-a442-129eea19ce6e\") " Sep 20 02:22:16.428550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428461 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home" (OuterVolumeSpecName: "home") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:22:16.428550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428536 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:22:16.428756 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:22:16.428802 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428749 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.428802 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.428769 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.430553 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.430533 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:22:16.430643 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.430581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm" (OuterVolumeSpecName: "dshm") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:22:16.430706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.430661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2" (OuterVolumeSpecName: "kube-api-access-s2pt2") pod "e222e765-bc76-47d8-a442-129eea19ce6e" (UID: "e222e765-bc76-47d8-a442-129eea19ce6e"). InnerVolumeSpecName "kube-api-access-s2pt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:22:16.529240 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.529214 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pt2\" (UniqueName: \"kubernetes.io/projected/e222e765-bc76-47d8-a442-129eea19ce6e-kube-api-access-s2pt2\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.529240 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.529239 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.529434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.529248 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e222e765-bc76-47d8-a442-129eea19ce6e-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.529434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.529257 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e222e765-bc76-47d8-a442-129eea19ce6e-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:22:16.532655 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.532629 2578 generic.go:356] "Generic (PLEG): container finished" podID="e222e765-bc76-47d8-a442-129eea19ce6e" containerID="783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d" exitCode=0 Sep 20 02:22:16.532766 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.532673 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" Sep 20 02:22:16.532766 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.532721 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" event={"ID":"e222e765-bc76-47d8-a442-129eea19ce6e","Type":"ContainerDied","Data":"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d"} Sep 20 02:22:16.532766 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.532762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn" event={"ID":"e222e765-bc76-47d8-a442-129eea19ce6e","Type":"ContainerDied","Data":"075170aa54a62d97c72169d049d65c24c8704111a5d4df551caf32e36c349a0e"} Sep 20 02:22:16.532870 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.532779 2578 scope.go:117] "RemoveContainer" containerID="783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d" Sep 20 02:22:16.545321 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.545306 2578 scope.go:117] "RemoveContainer" containerID="783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d" Sep 20 02:22:16.545599 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:22:16.545581 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d\": container with ID starting with 783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d not found: ID does not exist" containerID="783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d" Sep 20 02:22:16.545666 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.545606 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d"} err="failed to get container status \"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d\": rpc error: code = NotFound desc = could not find container \"783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d\": container with ID starting with 783ed6e3087f877e0b89a1a1958c05fa18d3563245dedf2031302d318838257d not found: ID does not exist" Sep 20 02:22:16.567019 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.566994 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:22:16.575185 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:16.575160 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-317780/auth-test-model-317780-kserve-595964d76c-s4grn"] Sep 20 02:22:17.015964 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:17.015925 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e222e765-bc76-47d8-a442-129eea19ce6e" path="/var/lib/kubelet/pods/e222e765-bc76-47d8-a442-129eea19ce6e/volumes" Sep 20 02:22:24.934360 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:24.934263 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:22:29.164534 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:29.164505 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:22:29.185180 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:29.185154 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:22:29.359057 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:29.359017 2578 scope.go:117] "RemoveContainer" containerID="5dfd55dca729a8dcd0a97997dc3dfcb807d280edfca29d8a9a3e5d8eecabe830" Sep 20 02:22:48.827353 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:22:48.827313 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:23:09.540986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:09.540949 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:23:22.086825 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.086793 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7534-8f053d"] Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087185 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e222e765-bc76-47d8-a442-129eea19ce6e" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087197 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e222e765-bc76-47d8-a442-129eea19ce6e" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087208 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087213 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087225 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49db703-24c3-4212-9656-bd0405ecf3ac" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087233 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49db703-24c3-4212-9656-bd0405ecf3ac" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087245 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087251 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087257 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087262 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087271 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087276 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087289 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087294 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087302 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" containerName="maas-api" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087307 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" containerName="maas-api" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087314 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadfd41c-4397-4982-a4e8-7346fac046e6" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087319 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadfd41c-4397-4982-a4e8-7346fac046e6" containerName="main" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087323 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064480f3-b97f-454d-a357-5578b334ba04" containerName="payload-pre-processing" Sep 20 02:23:22.087334 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087328 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="064480f3-b97f-454d-a357-5578b334ba04" containerName="payload-pre-processing" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087385 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c747e1-1d4d-4184-b5e4-b0206073eff1" containerName="main" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087391 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="064480f3-b97f-454d-a357-5578b334ba04" containerName="payload-pre-processing" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087397 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cadfd41c-4397-4982-a4e8-7346fac046e6" containerName="main" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087403 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087409 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d643321-8b1f-4c40-a1c3-7b1e5f66b0f3" containerName="maas-api" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087414 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba8cd0c-6a62-4c78-8506-4aea061aad6a" containerName="main" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087420 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087427 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf1355d-d8aa-4123-b5bc-25a9ff3ca87e" containerName="main" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087432 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e222e765-bc76-47d8-a442-129eea19ce6e" containerName="main" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087438 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087443 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:22.088021 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.087448 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49db703-24c3-4212-9656-bd0405ecf3ac" containerName="main" Sep 20 02:23:22.090502 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.090484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:22.099765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.099744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l75z\"" Sep 20 02:23:22.105352 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.105327 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-8f053d"] Sep 20 02:23:22.228363 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.228316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gd6s\" (UniqueName: \"kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s\") pod \"test-curl-7534-8f053d\" (UID: \"b6efc328-a271-413d-b85d-ba6cbb426f02\") " pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:22.329662 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.329619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gd6s\" (UniqueName: \"kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s\") pod \"test-curl-7534-8f053d\" (UID: \"b6efc328-a271-413d-b85d-ba6cbb426f02\") " pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:22.341968 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.341891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gd6s\" (UniqueName: \"kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s\") pod \"test-curl-7534-8f053d\" (UID: \"b6efc328-a271-413d-b85d-ba6cbb426f02\") " pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:22.399772 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.399735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:22.706617 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.706581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7534-8f053d"] Sep 20 02:23:22.708242 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:23:22.708218 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6efc328_a271_413d_b85d_ba6cbb426f02.slice/crio-7d3dee98fee52eb36e9f646fcd8455ff36965a598d8d37cbccaa3229f1a2330d WatchSource:0}: Error finding container 7d3dee98fee52eb36e9f646fcd8455ff36965a598d8d37cbccaa3229f1a2330d: Status 404 returned error can't find the container with id 7d3dee98fee52eb36e9f646fcd8455ff36965a598d8d37cbccaa3229f1a2330d Sep 20 02:23:22.914517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.914482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-8f053d" event={"ID":"b6efc328-a271-413d-b85d-ba6cbb426f02","Type":"ContainerStarted","Data":"0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf"} Sep 20 02:23:22.914704 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.914529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7534-8f053d" event={"ID":"b6efc328-a271-413d-b85d-ba6cbb426f02","Type":"ContainerStarted","Data":"7d3dee98fee52eb36e9f646fcd8455ff36965a598d8d37cbccaa3229f1a2330d"} Sep 20 02:23:22.942330 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:22.942267 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7534-8f053d" podStartSLOduration=0.942252265 podStartE2EDuration="942.252265ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:23:22.940106222 +0000 UTC m=+1854.458421981" watchObservedRunningTime="2026-09-20 02:23:22.942252265 +0000 UTC m=+1854.460568023" Sep 20 02:23:23.781325 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:23.781226 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7534-8f053d"] Sep 20 02:23:23.789829 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:23.789797 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7534-8f053d"] Sep 20 02:23:23.923060 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:23:23.923033 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7534-8f053d" container="test-curl-7534-8f053d" Sep 20 02:23:24.928493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:24.928445 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7534-8f053d" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" containerName="test-curl-7534-8f053d" containerID="cri-o://0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf" gracePeriod=2 Sep 20 02:23:24.940013 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:23:24.939979 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-8f053d\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:24.987074 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:24.987030 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:23:27.110689 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.110664 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:27.285497 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.284747 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gd6s\" (UniqueName: \"kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s\") pod \"b6efc328-a271-413d-b85d-ba6cbb426f02\" (UID: \"b6efc328-a271-413d-b85d-ba6cbb426f02\") " Sep 20 02:23:27.287639 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.287613 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s" (OuterVolumeSpecName: "kube-api-access-5gd6s") pod "b6efc328-a271-413d-b85d-ba6cbb426f02" (UID: "b6efc328-a271-413d-b85d-ba6cbb426f02"). InnerVolumeSpecName "kube-api-access-5gd6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:23:27.385706 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.385670 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gd6s\" (UniqueName: \"kubernetes.io/projected/b6efc328-a271-413d-b85d-ba6cbb426f02-kube-api-access-5gd6s\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:23:27.948203 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.948169 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6efc328-a271-413d-b85d-ba6cbb426f02" containerID="0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf" exitCode=137 Sep 20 02:23:27.948371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.948212 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:27.948371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.948265 2578 scope.go:117] "RemoveContainer" containerID="0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf" Sep 20 02:23:27.957612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.957592 2578 scope.go:117] "RemoveContainer" containerID="0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf" Sep 20 02:23:27.957896 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:23:27.957878 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf\": container with ID starting with 0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf not found: ID does not exist" containerID="0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf" Sep 20 02:23:27.958001 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:27.957920 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf"} err="failed to get container status \"0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf\": rpc error: code = NotFound desc = could not find container \"0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf\": container with ID starting with 0731185c96064b6c33c6fd2509f859524f592cd9edebebe02b34dee99f1c6ebf not found: ID does not exist" Sep 20 02:23:27.960962 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:23:27.960935 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-8f053d\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:29.014364 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:23:29.014326 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7534-8f053d\" is forbidden: User \"system:node:ip-10-0-137-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-50.ec2.internal' and this object" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" pod="opendatahub/test-curl-7534-8f053d" Sep 20 02:23:29.015392 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:29.015370 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" path="/var/lib/kubelet/pods/b6efc328-a271-413d-b85d-ba6cbb426f02/volumes" Sep 20 02:23:36.645302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:36.645266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:23:38.527345 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527312 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527684 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527695 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3f3829-13af-4e8a-b5fa-09cfc997ed36" containerName="istio-proxy" Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527733 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" containerName="test-curl-7534-8f053d" Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527739 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" containerName="test-curl-7534-8f053d" Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527753 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:38.527807 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527760 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b476887d-ae63-491c-9d46-4ab4cf2c67cf" containerName="istio-proxy" Sep 20 02:23:38.528159 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.527831 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6efc328-a271-413d-b85d-ba6cbb426f02" containerName="test-curl-7534-8f053d" Sep 20 02:23:38.530827 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.530807 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.535172 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.535150 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ed9f38\"/\"openshift-service-ca.crt\"" Sep 20 02:23:38.535928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.535893 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ed9f38\"/\"kube-root-ca.crt\"" Sep 20 02:23:38.535928 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.535920 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ed9f38\"/\"rate-test-model-ed9f38-kserve-self-signed-certs\"" Sep 20 02:23:38.536105 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.535950 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ed9f38\"/\"default-dockercfg-692d7\"" Sep 20 02:23:38.556440 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.556414 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:23:38.582222 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbqg\" (UniqueName: \"kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.582394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.582394 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.582500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.582500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.582500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.582486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683026 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.682984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683026 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683278 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683278 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbqg\" (UniqueName: \"kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683278 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683447 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683557 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.683671 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.683649 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.685460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.685443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.685730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.685712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.692480 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.692457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbqg\" (UniqueName: \"kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg\") pod \"rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:38.840342 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:38.840243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:39.125333 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:39.125306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:23:39.127561 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:23:39.127534 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9f67440_efa3_43c5_ad44_eb696e768576.slice/crio-20597caca8516a31e3e52e75f9615c8bb89a17aa8044b36ce5c9725d56b76c9c WatchSource:0}: Error finding container 20597caca8516a31e3e52e75f9615c8bb89a17aa8044b36ce5c9725d56b76c9c: Status 404 returned error can't find the container with id 20597caca8516a31e3e52e75f9615c8bb89a17aa8044b36ce5c9725d56b76c9c Sep 20 02:23:40.022986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:40.022941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" event={"ID":"d9f67440-efa3-43c5-ad44-eb696e768576","Type":"ContainerStarted","Data":"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1"} Sep 20 02:23:40.023471 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:40.022997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" event={"ID":"d9f67440-efa3-43c5-ad44-eb696e768576","Type":"ContainerStarted","Data":"20597caca8516a31e3e52e75f9615c8bb89a17aa8044b36ce5c9725d56b76c9c"} Sep 20 02:23:40.044763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:40.044715 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" podStartSLOduration=2.04470066 podStartE2EDuration="2.04470066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:23:40.043206728 +0000 UTC m=+1871.561522477" watchObservedRunningTime="2026-09-20 02:23:40.04470066 +0000 UTC m=+1871.563016709" Sep 20 02:23:42.791862 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:42.791827 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:23:48.840844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:48.840804 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:48.841259 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:48.840863 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:48.853342 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:48.853313 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:23:49.083958 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:23:49.083893 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:24:11.645954 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:11.645900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:24:25.143940 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:25.143876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:24:40.217171 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:40.217133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:24:46.004429 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:46.004394 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:24:50.890865 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:50.890826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:24:56.077223 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.077189 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:24:56.080980 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.080964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.087466 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.087446 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec8b82-dockercfg-lt978\"" Sep 20 02:24:56.087576 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.087448 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec8b82\"" Sep 20 02:24:56.107001 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.106974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:24:56.199322 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.199280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8wc\" (UniqueName: \"kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.199491 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.199338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.300453 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.300419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8wc\" (UniqueName: \"kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.300453 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.300457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.301198 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.301180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.316558 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.316526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8wc\" (UniqueName: \"kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc\") pod \"payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.391182 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.391152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:56.721763 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:56.721734 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:24:56.723960 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:24:56.723933 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd891689b_0450_45b7_9cf2_7f1ad3729e9b.slice/crio-4309039d18ecd336aaa87a0b9b5fde46e25ae6b78fb73dd62d9f520974651b97 WatchSource:0}: Error finding container 4309039d18ecd336aaa87a0b9b5fde46e25ae6b78fb73dd62d9f520974651b97: Status 404 returned error can't find the container with id 4309039d18ecd336aaa87a0b9b5fde46e25ae6b78fb73dd62d9f520974651b97 Sep 20 02:24:57.462383 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:57.462341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" event={"ID":"d891689b-0450-45b7-9cf2-7f1ad3729e9b","Type":"ContainerStarted","Data":"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783"} Sep 20 02:24:57.462774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:57.462412 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:24:57.462774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:57.462437 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" event={"ID":"d891689b-0450-45b7-9cf2-7f1ad3729e9b","Type":"ContainerStarted","Data":"4309039d18ecd336aaa87a0b9b5fde46e25ae6b78fb73dd62d9f520974651b97"} Sep 20 02:24:57.493537 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:24:57.493484 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" podStartSLOduration=1.4934623089999999 podStartE2EDuration="1.493462309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:24:57.492234768 +0000 UTC m=+1949.010550525" watchObservedRunningTime="2026-09-20 02:24:57.493462309 +0000 UTC m=+1949.011778068" Sep 20 02:25:01.023987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.023946 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" podUID="d9f67440-efa3-43c5-ad44-eb696e768576" containerName="main" containerID="cri-o://9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1" gracePeriod=45 Sep 20 02:25:01.272085 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.272062 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:25:01.349311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349230 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349269 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349311 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349307 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349324 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpbqg\" (UniqueName: \"kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349431 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache\") pod \"d9f67440-efa3-43c5-ad44-eb696e768576\" (UID: \"d9f67440-efa3-43c5-ad44-eb696e768576\") " Sep 20 02:25:01.349578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349546 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home" (OuterVolumeSpecName: "home") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:25:01.349792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349605 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:25:01.349792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349720 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.349792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349742 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.349792 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.349761 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:25:01.351719 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.351694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm" (OuterVolumeSpecName: "dshm") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:25:01.351820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.351749 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:25:01.351820 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.351762 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg" (OuterVolumeSpecName: "kube-api-access-jpbqg") pod "d9f67440-efa3-43c5-ad44-eb696e768576" (UID: "d9f67440-efa3-43c5-ad44-eb696e768576"). InnerVolumeSpecName "kube-api-access-jpbqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:25:01.451076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.451040 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.451076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.451073 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f67440-efa3-43c5-ad44-eb696e768576-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.451076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.451082 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f67440-efa3-43c5-ad44-eb696e768576-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.451076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.451092 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbqg\" (UniqueName: \"kubernetes.io/projected/d9f67440-efa3-43c5-ad44-eb696e768576-kube-api-access-jpbqg\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:25:01.488034 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.487997 2578 generic.go:356] "Generic (PLEG): container finished" podID="d9f67440-efa3-43c5-ad44-eb696e768576" containerID="9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1" exitCode=0 Sep 20 02:25:01.488205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.488046 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" Sep 20 02:25:01.488205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.488086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" event={"ID":"d9f67440-efa3-43c5-ad44-eb696e768576","Type":"ContainerDied","Data":"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1"} Sep 20 02:25:01.488205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.488141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc" event={"ID":"d9f67440-efa3-43c5-ad44-eb696e768576","Type":"ContainerDied","Data":"20597caca8516a31e3e52e75f9615c8bb89a17aa8044b36ce5c9725d56b76c9c"} Sep 20 02:25:01.488205 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.488164 2578 scope.go:117] "RemoveContainer" containerID="9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1" Sep 20 02:25:01.498529 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.498509 2578 scope.go:117] "RemoveContainer" containerID="9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1" Sep 20 02:25:01.498849 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:25:01.498823 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1\": container with ID starting with 9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1 not found: ID does not exist" containerID="9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1" Sep 20 02:25:01.498958 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.498862 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1"} err="failed to get container status \"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1\": rpc error: code = NotFound desc = could not find container \"9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1\": container with ID starting with 9d37839f4be6618a840acccdd9ca25785983a21494dd9cc85c6c163b7d4d13e1 not found: ID does not exist" Sep 20 02:25:01.519438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.519412 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:25:01.529067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:01.529045 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ed9f38/rate-test-model-ed9f38-kserve-68bb6f98f9-6hhtc"] Sep 20 02:25:03.016073 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:03.016039 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f67440-efa3-43c5-ad44-eb696e768576" path="/var/lib/kubelet/pods/d9f67440-efa3-43c5-ad44-eb696e768576/volumes" Sep 20 02:25:03.253984 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:03.253946 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:25:08.471740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:08.471707 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:25:30.089698 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.089661 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:25:30.090134 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.090124 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f67440-efa3-43c5-ad44-eb696e768576" containerName="main" Sep 20 02:25:30.090177 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.090138 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f67440-efa3-43c5-ad44-eb696e768576" containerName="main" Sep 20 02:25:30.090228 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.090217 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f67440-efa3-43c5-ad44-eb696e768576" containerName="main" Sep 20 02:25:30.093620 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.093603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.095957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.095933 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6be25a\"" Sep 20 02:25:30.096080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.095976 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6be25a-dockercfg-g6k7p\"" Sep 20 02:25:30.106371 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.106346 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:25:30.235993 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.235953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjbz\" (UniqueName: \"kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.236160 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.236004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.336900 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.336859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjbz\" (UniqueName: \"kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.337094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.336935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.337598 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.337576 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.346849 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.346781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjbz\" (UniqueName: \"kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz\") pod \"payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.403588 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.403552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:30.681087 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:30.681060 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:25:30.681825 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:25:30.681807 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fbde3e_2037_4760_8b3b_114d0cb2ee39.slice/crio-98df8fbfe252608c450c7b726fd68b41f7f2eb0c5cf398183d171b924388df1d WatchSource:0}: Error finding container 98df8fbfe252608c450c7b726fd68b41f7f2eb0c5cf398183d171b924388df1d: Status 404 returned error can't find the container with id 98df8fbfe252608c450c7b726fd68b41f7f2eb0c5cf398183d171b924388df1d Sep 20 02:25:31.663517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:31.663472 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" event={"ID":"68fbde3e-2037-4760-8b3b-114d0cb2ee39","Type":"ContainerStarted","Data":"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a"} Sep 20 02:25:31.663961 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:31.663545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" event={"ID":"68fbde3e-2037-4760-8b3b-114d0cb2ee39","Type":"ContainerStarted","Data":"98df8fbfe252608c450c7b726fd68b41f7f2eb0c5cf398183d171b924388df1d"} Sep 20 02:25:31.663961 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:31.663687 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:25:31.688296 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:31.688238 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" podStartSLOduration=1.688221666 podStartE2EDuration="1.688221666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:25:31.687035795 +0000 UTC m=+1983.205351557" watchObservedRunningTime="2026-09-20 02:25:31.688221666 +0000 UTC m=+1983.206537423" Sep 20 02:25:34.221071 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:34.221033 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:25:42.671312 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:25:42.671279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:26:00.383851 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:00.383819 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx_68fbde3e-2037-4760-8b3b-114d0cb2ee39/payload-processing/0.log" Sep 20 02:26:04.714604 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.714573 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:26:04.718239 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.718221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.721378 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.721303 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ec8b82\"/\"default-dockercfg-4v75n\"" Sep 20 02:26:04.721378 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.721312 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ec8b82\"/\"openshift-service-ca.crt\"" Sep 20 02:26:04.721378 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.721329 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ec8b82\"/\"ipp-route-ec8b82-kserve-self-signed-certs\"" Sep 20 02:26:04.721688 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.721668 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ec8b82\"/\"kube-root-ca.crt\"" Sep 20 02:26:04.729018 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.728992 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:26:04.830981 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.830943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.831161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.831014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.831161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.831048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.831161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.831074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.831161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.831140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88phv\" (UniqueName: \"kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.831302 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.831184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.932564 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.932790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.932790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.932790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.932790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88phv\" (UniqueName: \"kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.933119 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.932809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.933119 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.933028 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.933284 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.933192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.933284 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.933272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.935481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.935453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.935599 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.935588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:04.940941 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:04.940918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88phv\" (UniqueName: \"kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv\") pod \"ipp-route-ec8b82-kserve-58c49f97bc-f27vh\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:05.029080 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:05.029009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:05.305780 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:05.305748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:26:05.307588 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:26:05.307561 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb842e402_5c59_403d_8f04_1f72a759e45f.slice/crio-06a4c2bbc6a5f32ec5165ba3136f74cc75a3829ed703dab6f3a4a55a5d8a89e8 WatchSource:0}: Error finding container 06a4c2bbc6a5f32ec5165ba3136f74cc75a3829ed703dab6f3a4a55a5d8a89e8: Status 404 returned error can't find the container with id 06a4c2bbc6a5f32ec5165ba3136f74cc75a3829ed703dab6f3a4a55a5d8a89e8 Sep 20 02:26:05.856215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:05.856178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" event={"ID":"b842e402-5c59-403d-8f04-1f72a759e45f","Type":"ContainerStarted","Data":"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd"} Sep 20 02:26:05.856595 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:05.856230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" event={"ID":"b842e402-5c59-403d-8f04-1f72a759e45f","Type":"ContainerStarted","Data":"06a4c2bbc6a5f32ec5165ba3136f74cc75a3829ed703dab6f3a4a55a5d8a89e8"} Sep 20 02:26:05.875145 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:05.875102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" podStartSLOduration=1.875087566 podStartE2EDuration="1.875087566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:26:05.87199963 +0000 UTC m=+2017.390315389" watchObservedRunningTime="2026-09-20 02:26:05.875087566 +0000 UTC m=+2017.393403324" Sep 20 02:26:08.623157 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:08.623126 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:26:15.030079 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:15.030047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:15.030481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:15.030103 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:15.042767 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:15.042741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:15.926158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:15.926125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:26:25.017226 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:25.017192 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:26:31.599525 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:31.599495 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx_68fbde3e-2037-4760-8b3b-114d0cb2ee39/payload-processing/0.log" Sep 20 02:26:45.726962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:26:45.726921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:27:21.615617 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:21.615580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:27:29.208305 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:29.208267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:27:29.224047 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:29.224020 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:27:40.522653 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:40.522612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:27:43.021690 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.021652 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:27:43.022168 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.021849 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" podUID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" containerName="payload-pre-processing" containerID="cri-o://c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783" gracePeriod=30 Sep 20 02:27:43.269095 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.269071 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:27:43.373533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.373431 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw8wc\" (UniqueName: \"kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc\") pod \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " Sep 20 02:27:43.373533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.373510 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume\") pod \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\" (UID: \"d891689b-0450-45b7-9cf2-7f1ad3729e9b\") " Sep 20 02:27:43.373842 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.373818 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d891689b-0450-45b7-9cf2-7f1ad3729e9b" (UID: "d891689b-0450-45b7-9cf2-7f1ad3729e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:27:43.375633 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.375611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc" (OuterVolumeSpecName: "kube-api-access-nw8wc") pod "d891689b-0450-45b7-9cf2-7f1ad3729e9b" (UID: "d891689b-0450-45b7-9cf2-7f1ad3729e9b"). InnerVolumeSpecName "kube-api-access-nw8wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:27:43.410437 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.410408 2578 generic.go:356] "Generic (PLEG): container finished" podID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" containerID="c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783" exitCode=0 Sep 20 02:27:43.410615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.410453 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" Sep 20 02:27:43.410615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.410493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" event={"ID":"d891689b-0450-45b7-9cf2-7f1ad3729e9b","Type":"ContainerDied","Data":"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783"} Sep 20 02:27:43.410615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.410565 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp" event={"ID":"d891689b-0450-45b7-9cf2-7f1ad3729e9b","Type":"ContainerDied","Data":"4309039d18ecd336aaa87a0b9b5fde46e25ae6b78fb73dd62d9f520974651b97"} Sep 20 02:27:43.410615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.410588 2578 scope.go:117] "RemoveContainer" containerID="c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783" Sep 20 02:27:43.419738 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.419716 2578 scope.go:117] "RemoveContainer" containerID="c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783" Sep 20 02:27:43.420060 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:27:43.420041 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783\": container with ID starting with c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783 not found: ID does not exist" containerID="c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783" Sep 20 02:27:43.420121 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.420071 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783"} err="failed to get container status \"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783\": rpc error: code = NotFound desc = could not find container \"c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783\": container with ID starting with c2309c1d845a71ff2c61b7afca8711004894900c657b983110307409e96b6783 not found: ID does not exist" Sep 20 02:27:43.433236 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.433216 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:27:43.439104 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.439081 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec8b82-68fccb6c4-kcmwp"] Sep 20 02:27:43.474564 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.474539 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891689b-0450-45b7-9cf2-7f1ad3729e9b-plugins-config-volume\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:27:43.474659 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:43.474567 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8wc\" (UniqueName: \"kubernetes.io/projected/d891689b-0450-45b7-9cf2-7f1ad3729e9b-kube-api-access-nw8wc\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:27:45.016309 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:45.016273 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" path="/var/lib/kubelet/pods/d891689b-0450-45b7-9cf2-7f1ad3729e9b/volumes" Sep 20 02:27:48.568728 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:48.568692 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:27:54.520939 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:27:54.520253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:28:03.590975 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.590921 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" podUID="b842e402-5c59-403d-8f04-1f72a759e45f" containerName="main" containerID="cri-o://fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd" gracePeriod=45 Sep 20 02:28:03.836626 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.836601 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:28:03.966408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966380 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88phv\" (UniqueName: \"kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966446 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966478 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966494 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966525 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966579 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966576 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir\") pod \"b842e402-5c59-403d-8f04-1f72a759e45f\" (UID: \"b842e402-5c59-403d-8f04-1f72a759e45f\") " Sep 20 02:28:03.966822 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966736 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home" (OuterVolumeSpecName: "home") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:28:03.966885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966828 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:03.966885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966844 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:28:03.967010 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.966935 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:28:03.968898 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.968869 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:28:03.969170 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.969149 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm" (OuterVolumeSpecName: "dshm") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:28:03.969233 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:03.969183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv" (OuterVolumeSpecName: "kube-api-access-88phv") pod "b842e402-5c59-403d-8f04-1f72a759e45f" (UID: "b842e402-5c59-403d-8f04-1f72a759e45f"). InnerVolumeSpecName "kube-api-access-88phv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:28:04.068268 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.068232 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:04.068268 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.068266 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b842e402-5c59-403d-8f04-1f72a759e45f-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:04.068268 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.068275 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:04.068268 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.068283 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b842e402-5c59-403d-8f04-1f72a759e45f-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:04.068501 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.068291 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88phv\" (UniqueName: \"kubernetes.io/projected/b842e402-5c59-403d-8f04-1f72a759e45f-kube-api-access-88phv\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:04.532648 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.532615 2578 generic.go:356] "Generic (PLEG): container finished" podID="b842e402-5c59-403d-8f04-1f72a759e45f" containerID="fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd" exitCode=0 Sep 20 02:28:04.532810 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.532660 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" Sep 20 02:28:04.532810 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.532697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" event={"ID":"b842e402-5c59-403d-8f04-1f72a759e45f","Type":"ContainerDied","Data":"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd"} Sep 20 02:28:04.532810 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.532753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh" event={"ID":"b842e402-5c59-403d-8f04-1f72a759e45f","Type":"ContainerDied","Data":"06a4c2bbc6a5f32ec5165ba3136f74cc75a3829ed703dab6f3a4a55a5d8a89e8"} Sep 20 02:28:04.532810 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.532775 2578 scope.go:117] "RemoveContainer" containerID="fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd" Sep 20 02:28:04.542591 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.542571 2578 scope.go:117] "RemoveContainer" containerID="fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd" Sep 20 02:28:04.542857 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:28:04.542838 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd\": container with ID starting with fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd not found: ID does not exist" containerID="fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd" Sep 20 02:28:04.542899 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.542864 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd"} err="failed to get container status \"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd\": rpc error: code = NotFound desc = could not find container \"fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd\": container with ID starting with fe22061d3821ceb2b7c482580f232708b303706633f50c80cac44cd422fc59dd not found: ID does not exist" Sep 20 02:28:04.555961 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.555936 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:28:04.560757 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:04.560736 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ec8b82/ipp-route-ec8b82-kserve-58c49f97bc-f27vh"] Sep 20 02:28:05.015443 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:05.015408 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b842e402-5c59-403d-8f04-1f72a759e45f" path="/var/lib/kubelet/pods/b842e402-5c59-403d-8f04-1f72a759e45f/volumes" Sep 20 02:28:48.410509 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.410475 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:28:48.411069 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.410753 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" podUID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" containerName="payload-processing" containerID="cri-o://c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a" gracePeriod=30 Sep 20 02:28:48.661555 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.661489 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:28:48.778420 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.778388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume\") pod \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " Sep 20 02:28:48.778583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.778470 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltjbz\" (UniqueName: \"kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz\") pod \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\" (UID: \"68fbde3e-2037-4760-8b3b-114d0cb2ee39\") " Sep 20 02:28:48.778759 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.778737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68fbde3e-2037-4760-8b3b-114d0cb2ee39" (UID: "68fbde3e-2037-4760-8b3b-114d0cb2ee39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:28:48.780744 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.780719 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz" (OuterVolumeSpecName: "kube-api-access-ltjbz") pod "68fbde3e-2037-4760-8b3b-114d0cb2ee39" (UID: "68fbde3e-2037-4760-8b3b-114d0cb2ee39"). InnerVolumeSpecName "kube-api-access-ltjbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:28:48.790575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.790550 2578 generic.go:356] "Generic (PLEG): container finished" podID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" containerID="c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a" exitCode=0 Sep 20 02:28:48.790676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.790589 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" Sep 20 02:28:48.790676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.790627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" event={"ID":"68fbde3e-2037-4760-8b3b-114d0cb2ee39","Type":"ContainerDied","Data":"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a"} Sep 20 02:28:48.790765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.790677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx" event={"ID":"68fbde3e-2037-4760-8b3b-114d0cb2ee39","Type":"ContainerDied","Data":"98df8fbfe252608c450c7b726fd68b41f7f2eb0c5cf398183d171b924388df1d"} Sep 20 02:28:48.790765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.790702 2578 scope.go:117] "RemoveContainer" containerID="c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a" Sep 20 02:28:48.800354 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.800336 2578 scope.go:117] "RemoveContainer" containerID="c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a" Sep 20 02:28:48.800618 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:28:48.800598 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a\": container with ID starting with c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a not found: ID does not exist" containerID="c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a" Sep 20 02:28:48.800681 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.800627 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a"} err="failed to get container status \"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a\": rpc error: code = NotFound desc = could not find container \"c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a\": container with ID starting with c8caa1b61d01e9ab41ca33e6c03426ca147d82e06c74b0cb7956e023d834a02a not found: ID does not exist" Sep 20 02:28:48.811136 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.811111 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:28:48.814785 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.814765 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6be25a-c6fb865df-txgvx"] Sep 20 02:28:48.879317 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.879292 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltjbz\" (UniqueName: \"kubernetes.io/projected/68fbde3e-2037-4760-8b3b-114d0cb2ee39-kube-api-access-ltjbz\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:48.879317 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:48.879316 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbde3e-2037-4760-8b3b-114d0cb2ee39-plugins-config-volume\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:28:49.015285 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:49.015193 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" path="/var/lib/kubelet/pods/68fbde3e-2037-4760-8b3b-114d0cb2ee39/volumes" Sep 20 02:28:51.849919 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:28:51.849870 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:29:06.255731 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.255691 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:29:06.256612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.256525 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" podUID="16e68632-9961-466c-a132-582d8a3d1519" containerName="maas-api" containerID="cri-o://377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10" gracePeriod=30 Sep 20 02:29:06.521507 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.521431 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:29:06.655580 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.655547 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4x57\" (UniqueName: \"kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57\") pod \"16e68632-9961-466c-a132-582d8a3d1519\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " Sep 20 02:29:06.655580 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.655583 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca\") pod \"16e68632-9961-466c-a132-582d8a3d1519\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " Sep 20 02:29:06.655790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.655704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs\") pod \"16e68632-9961-466c-a132-582d8a3d1519\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " Sep 20 02:29:06.655790 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.655757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls\") pod \"16e68632-9961-466c-a132-582d8a3d1519\" (UID: \"16e68632-9961-466c-a132-582d8a3d1519\") " Sep 20 02:29:06.656005 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.655980 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16e68632-9961-466c-a132-582d8a3d1519" (UID: "16e68632-9961-466c-a132-582d8a3d1519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:29:06.656207 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.656186 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e68632-9961-466c-a132-582d8a3d1519-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:06.658018 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.657990 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57" (OuterVolumeSpecName: "kube-api-access-m4x57") pod "16e68632-9961-466c-a132-582d8a3d1519" (UID: "16e68632-9961-466c-a132-582d8a3d1519"). InnerVolumeSpecName "kube-api-access-m4x57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:29:06.658018 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.658006 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16e68632-9961-466c-a132-582d8a3d1519" (UID: "16e68632-9961-466c-a132-582d8a3d1519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:29:06.658151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.658082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16e68632-9961-466c-a132-582d8a3d1519" (UID: "16e68632-9961-466c-a132-582d8a3d1519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:29:06.756760 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.756722 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:06.756760 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.756755 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e68632-9961-466c-a132-582d8a3d1519-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:06.756760 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.756765 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4x57\" (UniqueName: \"kubernetes.io/projected/16e68632-9961-466c-a132-582d8a3d1519-kube-api-access-m4x57\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:06.895605 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.895568 2578 generic.go:356] "Generic (PLEG): container finished" podID="16e68632-9961-466c-a132-582d8a3d1519" containerID="377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10" exitCode=0 Sep 20 02:29:06.895765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.895613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" event={"ID":"16e68632-9961-466c-a132-582d8a3d1519","Type":"ContainerDied","Data":"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10"} Sep 20 02:29:06.895765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.895677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" event={"ID":"16e68632-9961-466c-a132-582d8a3d1519","Type":"ContainerDied","Data":"722f3440c87db3de692cd57d6630a680e7c67717950e83e3546b29ce367224d9"} Sep 20 02:29:06.895765 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.895613 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v" Sep 20 02:29:06.895890 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.895698 2578 scope.go:117] "RemoveContainer" containerID="377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10" Sep 20 02:29:06.906117 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.906098 2578 scope.go:117] "RemoveContainer" containerID="377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10" Sep 20 02:29:06.906462 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:29:06.906443 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10\": container with ID starting with 377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10 not found: ID does not exist" containerID="377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10" Sep 20 02:29:06.906521 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.906473 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10"} err="failed to get container status \"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10\": rpc error: code = NotFound desc = could not find container \"377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10\": container with ID starting with 377401af0b5d685389a522ea82e3834a4742b4e716de4a1040e63e04eda3ad10 not found: ID does not exist" Sep 20 02:29:06.919349 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.919320 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:29:06.927312 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:06.927284 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ed9f38-658d49dfbc-7mr6v"] Sep 20 02:29:07.016773 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:07.016738 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e68632-9961-466c-a132-582d8a3d1519" path="/var/lib/kubelet/pods/16e68632-9961-466c-a132-582d8a3d1519/volumes" Sep 20 02:29:09.466100 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:09.466059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:29:13.559473 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:13.559427 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:29:13.559953 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:13.559657 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" containerID="cri-o://0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a" gracePeriod=30 Sep 20 02:29:17.205276 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:17.205232 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 20 02:29:17.205662 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:17.205302 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 20 02:29:18.817265 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.817238 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:29:18.850657 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850580 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850657 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850646 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850678 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850723 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ztzl\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.850885 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.851245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850883 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:29:18.851245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.850921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data\") pod \"39924e1e-ff2a-48c6-85d3-82334ce668dc\" (UID: \"39924e1e-ff2a-48c6-85d3-82334ce668dc\") " Sep 20 02:29:18.851245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851051 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:18.851245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851076 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:18.851458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851296 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39924e1e-ff2a-48c6-85d3-82334ce668dc-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.851458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851318 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.851458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851334 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.851458 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:18.851667 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.851553 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:18.853233 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.853206 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:29:18.853370 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.853325 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl" (OuterVolumeSpecName: "kube-api-access-2ztzl") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "kube-api-access-2ztzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:29:18.853627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.853605 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:29:18.853627 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.853611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39924e1e-ff2a-48c6-85d3-82334ce668dc" (UID: "39924e1e-ff2a-48c6-85d3-82334ce668dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:18.951732 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951700 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.951942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951742 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.951942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951751 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztzl\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-kube-api-access-2ztzl\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.951942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951760 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.951942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951769 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.951942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.951776 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39924e1e-ff2a-48c6-85d3-82334ce668dc-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:18.974165 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.974139 2578 generic.go:356] "Generic (PLEG): container finished" podID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerID="0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a" exitCode=0 Sep 20 02:29:18.974300 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.974189 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" Sep 20 02:29:18.974300 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.974228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerDied","Data":"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a"} Sep 20 02:29:18.974300 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.974288 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x" event={"ID":"39924e1e-ff2a-48c6-85d3-82334ce668dc","Type":"ContainerDied","Data":"9d5452a7f160bafcab37b1653af83872054c427d37a15240dc7bc54fee4a0115"} Sep 20 02:29:18.974432 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.974318 2578 scope.go:117] "RemoveContainer" containerID="0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a" Sep 20 02:29:18.984688 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.984665 2578 scope.go:117] "RemoveContainer" containerID="5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de" Sep 20 02:29:18.993844 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.993824 2578 scope.go:117] "RemoveContainer" containerID="0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a" Sep 20 02:29:18.994182 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:29:18.994163 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a\": container with ID starting with 0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a not found: ID does not exist" containerID="0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a" Sep 20 02:29:18.994266 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.994190 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a"} err="failed to get container status \"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a\": rpc error: code = NotFound desc = could not find container \"0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a\": container with ID starting with 0e4166d1e3d47dea39be7dfcb971204856afc2ab81c130884517f86cddd6193a not found: ID does not exist" Sep 20 02:29:18.994266 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.994209 2578 scope.go:117] "RemoveContainer" containerID="5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de" Sep 20 02:29:18.994476 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:29:18.994460 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de\": container with ID starting with 5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de not found: ID does not exist" containerID="5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de" Sep 20 02:29:18.994520 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.994482 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de"} err="failed to get container status \"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de\": rpc error: code = NotFound desc = could not find container \"5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de\": container with ID starting with 5f77205d67f51184bcec8c22a8298c141d848f26aaedbc0d01696ed5ffa8a7de not found: ID does not exist" Sep 20 02:29:18.997767 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:18.997744 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:29:19.004453 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:19.004432 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ed9f38-openshift-default-8dd7cf4-ps78x"] Sep 20 02:29:19.016733 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:19.016704 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" path="/var/lib/kubelet/pods/39924e1e-ff2a-48c6-85d3-82334ce668dc/volumes" Sep 20 02:29:27.435867 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:27.435826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:29:32.980461 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:32.980420 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:29:32.980985 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:32.980776 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" containerID="cri-o://26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1" gracePeriod=30 Sep 20 02:29:35.493396 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:35.493359 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 20 02:29:35.493834 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:35.493436 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 20 02:29:37.115987 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:37.115953 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:29:38.220612 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.220589 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:29:38.330511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlcmd\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330505 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330533 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330571 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330603 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.330774 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330686 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs\") pod \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\" (UID: \"99df1ec9-cd16-401d-bdf9-4b69ff91e58f\") " Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330876 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data" (OuterVolumeSpecName: "istio-data") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.330979 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.331051 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-credential-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.331111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.331100 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-socket\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.331408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.331117 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-data\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.331408 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.331141 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:38.332962 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.332892 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd" (OuterVolumeSpecName: "kube-api-access-zlcmd") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "kube-api-access-zlcmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:29:38.333096 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.333077 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token" (OuterVolumeSpecName: "istio-token") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:29:38.333161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.333106 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:29:38.333161 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.333137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99df1ec9-cd16-401d-bdf9-4b69ff91e58f" (UID: "99df1ec9-cd16-401d-bdf9-4b69ff91e58f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 20 02:29:38.431896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431850 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istiod-ca-cert\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.431896 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431892 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlcmd\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-kube-api-access-zlcmd\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.432094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431933 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-podinfo\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.432094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431948 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-envoy\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.432094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431959 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-istio-token\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:38.432094 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:38.431969 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99df1ec9-cd16-401d-bdf9-4b69ff91e58f-workload-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:29:39.084961 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.084926 2578 generic.go:356] "Generic (PLEG): container finished" podID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerID="26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1" exitCode=0 Sep 20 02:29:39.085115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.084984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" Sep 20 02:29:39.085115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.085006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerDied","Data":"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1"} Sep 20 02:29:39.085115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.085057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8" event={"ID":"99df1ec9-cd16-401d-bdf9-4b69ff91e58f","Type":"ContainerDied","Data":"81a153821fbe7d4ebaec36ca7da51d146d48ca935b8054acad41d448e88770bd"} Sep 20 02:29:39.085115 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.085080 2578 scope.go:117] "RemoveContainer" containerID="26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1" Sep 20 02:29:39.094206 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.094186 2578 scope.go:117] "RemoveContainer" containerID="d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35" Sep 20 02:29:39.103037 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.102860 2578 scope.go:117] "RemoveContainer" containerID="26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1" Sep 20 02:29:39.103236 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:29:39.103215 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1\": container with ID starting with 26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1 not found: ID does not exist" containerID="26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1" Sep 20 02:29:39.103288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.103244 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1"} err="failed to get container status \"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1\": rpc error: code = NotFound desc = could not find container \"26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1\": container with ID starting with 26578a3ea7fb23e0fe813a93b930a69471b0689b60ab4ed9c829b2d832dc2bc1 not found: ID does not exist" Sep 20 02:29:39.103288 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.103263 2578 scope.go:117] "RemoveContainer" containerID="d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35" Sep 20 02:29:39.103530 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:29:39.103512 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35\": container with ID starting with d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35 not found: ID does not exist" containerID="d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35" Sep 20 02:29:39.103575 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.103536 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35"} err="failed to get container status \"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35\": rpc error: code = NotFound desc = could not find container \"d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35\": container with ID starting with d8332d241f13dec579ed28db5bb8439463b7d64c89844807e2d8d51cd9664e35 not found: ID does not exist" Sep 20 02:29:39.109725 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.109702 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:29:39.121285 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:39.121261 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-317780-openshift-default-794cb8f58b-kw4t8"] Sep 20 02:29:41.015583 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:29:41.015548 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" path="/var/lib/kubelet/pods/99df1ec9-cd16-401d-bdf9-4b69ff91e58f/volumes" Sep 20 02:30:00.143076 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.142992 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq"] Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143363 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e68632-9961-466c-a132-582d8a3d1519" containerName="maas-api" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143374 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e68632-9961-466c-a132-582d8a3d1519" containerName="maas-api" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143384 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143390 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143405 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" containerName="payload-processing" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143410 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" containerName="payload-processing" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143418 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143424 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143432 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143437 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143446 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143452 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143463 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b842e402-5c59-403d-8f04-1f72a759e45f" containerName="main" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143468 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b842e402-5c59-403d-8f04-1f72a759e45f" containerName="main" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143475 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" containerName="payload-pre-processing" Sep 20 02:30:00.143517 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143480 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" containerName="payload-pre-processing" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143531 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e68632-9961-466c-a132-582d8a3d1519" containerName="maas-api" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143539 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b842e402-5c59-403d-8f04-1f72a759e45f" containerName="main" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143555 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143562 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fbde3e-2037-4760-8b3b-114d0cb2ee39" containerName="payload-processing" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143570 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143575 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d891689b-0450-45b7-9cf2-7f1ad3729e9b" containerName="payload-pre-processing" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143581 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39924e1e-ff2a-48c6-85d3-82334ce668dc" containerName="istio-proxy" Sep 20 02:30:00.144141 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.143585 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99df1ec9-cd16-401d-bdf9-4b69ff91e58f" containerName="istio-proxy" Sep 20 02:30:00.148190 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.148173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" Sep 20 02:30:00.151333 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.151283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5gscp\"" Sep 20 02:30:00.154159 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.154132 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq"] Sep 20 02:30:00.159188 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.159167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" Sep 20 02:30:00.440064 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:00.440038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq"] Sep 20 02:30:00.441680 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:30:00.441648 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e2be10_e0b8_4660_8be3_8bc9c35061b1.slice/crio-18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45 WatchSource:0}: Error finding container 18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45: Status 404 returned error can't find the container with id 18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45 Sep 20 02:30:01.208095 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:01.208061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" event={"ID":"e9e2be10-e0b8-4660-8be3-8bc9c35061b1","Type":"ContainerStarted","Data":"b9065d33bcb7d1d4240bc941efe7c7ce01b4d68679d93a0c57b0585ad4627edc"} Sep 20 02:30:01.208569 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:01.208123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" event={"ID":"e9e2be10-e0b8-4660-8be3-8bc9c35061b1","Type":"ContainerStarted","Data":"18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45"} Sep 20 02:30:01.225668 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:01.225618 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" podStartSLOduration=1.225600763 podStartE2EDuration="1.225600763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:30:01.223585728 +0000 UTC m=+2252.741901486" watchObservedRunningTime="2026-09-20 02:30:01.225600763 +0000 UTC m=+2252.743916532" Sep 20 02:30:02.215149 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:02.215115 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9e2be10-e0b8-4660-8be3-8bc9c35061b1" containerID="b9065d33bcb7d1d4240bc941efe7c7ce01b4d68679d93a0c57b0585ad4627edc" exitCode=0 Sep 20 02:30:02.215524 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:02.215204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" event={"ID":"e9e2be10-e0b8-4660-8be3-8bc9c35061b1","Type":"ContainerDied","Data":"b9065d33bcb7d1d4240bc941efe7c7ce01b4d68679d93a0c57b0585ad4627edc"} Sep 20 02:30:03.350388 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:03.350364 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" Sep 20 02:30:04.227364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:04.227335 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" Sep 20 02:30:04.227364 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:04.227345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29831190-fscfq" event={"ID":"e9e2be10-e0b8-4660-8be3-8bc9c35061b1","Type":"ContainerDied","Data":"18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45"} Sep 20 02:30:04.227567 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:04.227401 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fe5de4f9246b1f2d807d1140ca484eaf6ff273795681c408cfc6734c860e45" Sep 20 02:30:57.712961 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:30:57.712922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:31:37.733126 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:31:37.733049 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:31:46.115569 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:31:46.115529 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:31:53.118874 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:31:53.118829 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:32:00.226572 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:00.226541 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:32:09.212881 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:09.212844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:32:17.422351 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:17.422310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:32:25.816336 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:25.816299 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:32:29.250715 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:29.250685 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:32:29.261025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:29.260997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7b7rr_1a6c450e-f2fc-44cd-9e77-caf329de70e7/ovn-acl-logging/0.log" Sep 20 02:32:33.727573 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:32:33.727538 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:34:02.119954 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:02.119891 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:34:08.822592 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:08.822551 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:34:18.225965 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.225921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:34:18.822986 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.822949 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:34:18.823384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.823369 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e2be10-e0b8-4660-8be3-8bc9c35061b1" containerName="cleanup" Sep 20 02:34:18.823384 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.823383 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e2be10-e0b8-4660-8be3-8bc9c35061b1" containerName="cleanup" Sep 20 02:34:18.823515 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.823452 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e2be10-e0b8-4660-8be3-8bc9c35061b1" containerName="cleanup" Sep 20 02:34:18.826578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.826555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.830013 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.829990 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5ce30569-kserve-self-signed-certs\"" Sep 20 02:34:18.840919 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.840874 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:34:18.940350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.940350 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.940578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.940578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.940578 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trtq\" (UniqueName: \"kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:18.940697 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:18.940586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041199 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trtq\" (UniqueName: \"kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041416 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041697 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041697 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.041768 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.041719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.043846 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.043827 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.044041 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.044023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.050500 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.050475 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trtq\" (UniqueName: \"kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq\") pod \"e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.135751 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.135641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:19.419869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.419842 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:34:19.422410 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:34:19.422382 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99ba6dc9_34cc_437a_ab8e_f8770adaeef9.slice/crio-e50673bb23e36aa660a5cb7d1d0f6fad0e67c74d87023c37b3a532bf02cfa5cf WatchSource:0}: Error finding container e50673bb23e36aa660a5cb7d1d0f6fad0e67c74d87023c37b3a532bf02cfa5cf: Status 404 returned error can't find the container with id e50673bb23e36aa660a5cb7d1d0f6fad0e67c74d87023c37b3a532bf02cfa5cf Sep 20 02:34:19.642722 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.642687 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" event={"ID":"99ba6dc9-34cc-437a-ab8e-f8770adaeef9","Type":"ContainerStarted","Data":"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995"} Sep 20 02:34:19.642934 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.642733 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" event={"ID":"99ba6dc9-34cc-437a-ab8e-f8770adaeef9","Type":"ContainerStarted","Data":"e50673bb23e36aa660a5cb7d1d0f6fad0e67c74d87023c37b3a532bf02cfa5cf"} Sep 20 02:34:19.663927 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:19.663850 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" podStartSLOduration=1.6638330319999999 podStartE2EDuration="1.663833032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:34:19.661309335 +0000 UTC m=+2511.179625092" watchObservedRunningTime="2026-09-20 02:34:19.663833032 +0000 UTC m=+2511.182148790" Sep 20 02:34:27.927481 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:27.927388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:34:29.136811 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:29.136762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:29.137303 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:29.136842 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:29.149438 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:29.149402 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:29.712489 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:29.712452 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:34:45.307560 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:45.307514 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:34:48.528634 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:34:48.528596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:00.328651 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.328586 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" podUID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" containerName="main" containerID="cri-o://ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995" gracePeriod=45 Sep 20 02:35:00.574345 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.574320 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:35:00.627676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627586 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.627676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.627676 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627669 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.628002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627723 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6trtq\" (UniqueName: \"kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.628002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627781 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.628002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627809 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm\") pod \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\" (UID: \"99ba6dc9-34cc-437a-ab8e-f8770adaeef9\") " Sep 20 02:35:00.628002 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.627958 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache" (OuterVolumeSpecName: "model-cache") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:35:00.628197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.628003 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:35:00.628197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.628080 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home" (OuterVolumeSpecName: "home") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:35:00.628197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.628132 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tmp-dir\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.628197 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.628150 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-model-cache\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.630198 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.630168 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:35:00.630321 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.630203 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq" (OuterVolumeSpecName: "kube-api-access-6trtq") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "kube-api-access-6trtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:35:00.630321 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.630255 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm" (OuterVolumeSpecName: "dshm") pod "99ba6dc9-34cc-437a-ab8e-f8770adaeef9" (UID: "99ba6dc9-34cc-437a-ab8e-f8770adaeef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 20 02:35:00.729550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.729509 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-tls-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.729550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.729544 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6trtq\" (UniqueName: \"kubernetes.io/projected/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-kube-api-access-6trtq\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.729550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.729555 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-home\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.729796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.729566 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ba6dc9-34cc-437a-ab8e-f8770adaeef9-dshm\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:00.870550 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.870513 2578 generic.go:356] "Generic (PLEG): container finished" podID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" containerID="ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995" exitCode=0 Sep 20 02:35:00.870739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.870565 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" Sep 20 02:35:00.870739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.870603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" event={"ID":"99ba6dc9-34cc-437a-ab8e-f8770adaeef9","Type":"ContainerDied","Data":"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995"} Sep 20 02:35:00.870739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.870648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9" event={"ID":"99ba6dc9-34cc-437a-ab8e-f8770adaeef9","Type":"ContainerDied","Data":"e50673bb23e36aa660a5cb7d1d0f6fad0e67c74d87023c37b3a532bf02cfa5cf"} Sep 20 02:35:00.870739 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.870668 2578 scope.go:117] "RemoveContainer" containerID="ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995" Sep 20 02:35:00.880245 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.880226 2578 scope.go:117] "RemoveContainer" containerID="ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995" Sep 20 02:35:00.880539 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:35:00.880519 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995\": container with ID starting with ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995 not found: ID does not exist" containerID="ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995" Sep 20 02:35:00.880585 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.880548 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995"} err="failed to get container status \"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995\": rpc error: code = NotFound desc = could not find container \"ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995\": container with ID starting with ffa197c7bfd40c3e5afd84d81afeb885f7972580a8f8ae7c81fb4327368f7995 not found: ID does not exist" Sep 20 02:35:00.893951 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.893924 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:35:00.899348 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:00.899314 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5ce30569-kserve-84fbfd4fdc-gqdj9"] Sep 20 02:35:01.016163 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:01.016130 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" path="/var/lib/kubelet/pods/99ba6dc9-34cc-437a-ab8e-f8770adaeef9/volumes" Sep 20 02:35:01.228818 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:01.228781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:08.129236 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.129198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:08.520516 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.520480 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:08.520887 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.520873 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" containerName="main" Sep 20 02:35:08.520974 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.520888 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" containerName="main" Sep 20 02:35:08.521025 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.521015 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99ba6dc9-34cc-437a-ab8e-f8770adaeef9" containerName="main" Sep 20 02:35:08.525496 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.525435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.530552 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.530527 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 20 02:35:08.530806 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.530783 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 20 02:35:08.531390 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.531366 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 20 02:35:08.531498 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.531429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 20 02:35:08.531498 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.531448 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z4spz\"" Sep 20 02:35:08.535955 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.535439 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:08.599857 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.599825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.600033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.599870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnvn\" (UniqueName: \"kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.600033 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.599993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.600120 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.600034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.700864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.700821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.700864 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.700875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.701151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.700932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.701151 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.700959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnvn\" (UniqueName: \"kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.701493 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.701469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.703434 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.703408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.703637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.703616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.708317 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.708295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnvn\" (UniqueName: \"kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn\") pod \"maas-api-5747fd5c8c-cj4wn\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:08.840380 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:08.840289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:09.112471 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.112443 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:09.114071 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:35:09.114045 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe434eba_c054_4649_8ef2_67d323b1fb41.slice/crio-549134ecca370a781119d591f6eb05ef3be0255fb6045ed07f14e93e264ecca9 WatchSource:0}: Error finding container 549134ecca370a781119d591f6eb05ef3be0255fb6045ed07f14e93e264ecca9: Status 404 returned error can't find the container with id 549134ecca370a781119d591f6eb05ef3be0255fb6045ed07f14e93e264ecca9 Sep 20 02:35:09.115705 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.115687 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 20 02:35:09.924826 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.924788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" event={"ID":"fe434eba-c054-4649-8ef2-67d323b1fb41","Type":"ContainerStarted","Data":"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6"} Sep 20 02:35:09.925235 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.924841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" event={"ID":"fe434eba-c054-4649-8ef2-67d323b1fb41","Type":"ContainerStarted","Data":"549134ecca370a781119d591f6eb05ef3be0255fb6045ed07f14e93e264ecca9"} Sep 20 02:35:09.925235 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.924950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:09.944307 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:09.944253 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" podStartSLOduration=1.453805759 podStartE2EDuration="1.944234999s" totalImagesPullingTime="490.42924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:35:09.942385292 +0000 UTC m=+2561.460701050" watchObservedRunningTime="2026-09-20 02:35:09.944234999 +0000 UTC m=+2561.462550758" Sep 20 02:35:15.902162 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.902121 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:15.902678 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.902353 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" podUID="fe434eba-c054-4649-8ef2-67d323b1fb41" containerName="maas-api" containerID="cri-o://66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6" gracePeriod=30 Sep 20 02:35:15.907067 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.907043 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:15.944338 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.944308 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58647f984c-9ghkw"] Sep 20 02:35:15.947805 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.947788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:15.958454 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:15.958426 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58647f984c-9ghkw"] Sep 20 02:35:16.067153 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.067121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0524642c-116d-40f4-b0e1-c78a0885146d-openshift-service-ca\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.067426 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.067212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-maas-api-tls\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.067426 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.067249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-metrics-certs\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.067426 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.067277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrqd\" (UniqueName: \"kubernetes.io/projected/0524642c-116d-40f4-b0e1-c78a0885146d-kube-api-access-bwrqd\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.126030 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.125997 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:16.147319 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.147291 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:16.168127 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.168057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-maas-api-tls\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.168274 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.168137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-metrics-certs\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.168274 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.168204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrqd\" (UniqueName: \"kubernetes.io/projected/0524642c-116d-40f4-b0e1-c78a0885146d-kube-api-access-bwrqd\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.168274 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.168244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0524642c-116d-40f4-b0e1-c78a0885146d-openshift-service-ca\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.169861 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.169836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0524642c-116d-40f4-b0e1-c78a0885146d-openshift-service-ca\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.171841 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.171819 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-maas-api-tls\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.171990 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.171876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0524642c-116d-40f4-b0e1-c78a0885146d-metrics-certs\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.178112 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.178092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrqd\" (UniqueName: \"kubernetes.io/projected/0524642c-116d-40f4-b0e1-c78a0885146d-kube-api-access-bwrqd\") pod \"maas-api-58647f984c-9ghkw\" (UID: \"0524642c-116d-40f4-b0e1-c78a0885146d\") " pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.269635 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.269609 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca\") pod \"fe434eba-c054-4649-8ef2-67d323b1fb41\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " Sep 20 02:35:16.269796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.269654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls\") pod \"fe434eba-c054-4649-8ef2-67d323b1fb41\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " Sep 20 02:35:16.269796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.269708 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jnvn\" (UniqueName: \"kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn\") pod \"fe434eba-c054-4649-8ef2-67d323b1fb41\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " Sep 20 02:35:16.269796 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.269767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs\") pod \"fe434eba-c054-4649-8ef2-67d323b1fb41\" (UID: \"fe434eba-c054-4649-8ef2-67d323b1fb41\") " Sep 20 02:35:16.270064 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.270035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe434eba-c054-4649-8ef2-67d323b1fb41" (UID: "fe434eba-c054-4649-8ef2-67d323b1fb41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 20 02:35:16.271213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.271188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:16.271981 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.271960 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe434eba-c054-4649-8ef2-67d323b1fb41" (UID: "fe434eba-c054-4649-8ef2-67d323b1fb41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:35:16.272156 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.272138 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn" (OuterVolumeSpecName: "kube-api-access-8jnvn") pod "fe434eba-c054-4649-8ef2-67d323b1fb41" (UID: "fe434eba-c054-4649-8ef2-67d323b1fb41"). InnerVolumeSpecName "kube-api-access-8jnvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 20 02:35:16.272227 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.272195 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe434eba-c054-4649-8ef2-67d323b1fb41" (UID: "fe434eba-c054-4649-8ef2-67d323b1fb41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 20 02:35:16.370869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.370833 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-metrics-certs\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:16.370869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.370863 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe434eba-c054-4649-8ef2-67d323b1fb41-openshift-service-ca\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:16.370869 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.370873 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe434eba-c054-4649-8ef2-67d323b1fb41-maas-api-tls\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:16.371105 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.370883 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jnvn\" (UniqueName: \"kubernetes.io/projected/fe434eba-c054-4649-8ef2-67d323b1fb41-kube-api-access-8jnvn\") on node \"ip-10-0-137-50.ec2.internal\" DevicePath \"\"" Sep 20 02:35:16.547183 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.547156 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58647f984c-9ghkw"] Sep 20 02:35:16.548326 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:35:16.548295 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0524642c_116d_40f4_b0e1_c78a0885146d.slice/crio-13c23d52609bd0da43321cb750e245e7208ba116fcecbee981a71a96ca2ad1fb WatchSource:0}: Error finding container 13c23d52609bd0da43321cb750e245e7208ba116fcecbee981a71a96ca2ad1fb: Status 404 returned error can't find the container with id 13c23d52609bd0da43321cb750e245e7208ba116fcecbee981a71a96ca2ad1fb Sep 20 02:35:16.967966 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.967935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" event={"ID":"0524642c-116d-40f4-b0e1-c78a0885146d","Type":"ContainerStarted","Data":"13c23d52609bd0da43321cb750e245e7208ba116fcecbee981a71a96ca2ad1fb"} Sep 20 02:35:16.969494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.969467 2578 generic.go:356] "Generic (PLEG): container finished" podID="fe434eba-c054-4649-8ef2-67d323b1fb41" containerID="66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6" exitCode=0 Sep 20 02:35:16.969592 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.969502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" event={"ID":"fe434eba-c054-4649-8ef2-67d323b1fb41","Type":"ContainerDied","Data":"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6"} Sep 20 02:35:16.969592 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.969512 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" Sep 20 02:35:16.969592 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.969538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn" event={"ID":"fe434eba-c054-4649-8ef2-67d323b1fb41","Type":"ContainerDied","Data":"549134ecca370a781119d591f6eb05ef3be0255fb6045ed07f14e93e264ecca9"} Sep 20 02:35:16.969592 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.969558 2578 scope.go:117] "RemoveContainer" containerID="66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6" Sep 20 02:35:16.986298 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.986279 2578 scope.go:117] "RemoveContainer" containerID="66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6" Sep 20 02:35:16.986595 ip-10-0-137-50 kubenswrapper[2578]: E0920 02:35:16.986579 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6\": container with ID starting with 66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6 not found: ID does not exist" containerID="66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6" Sep 20 02:35:16.986637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:16.986605 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6"} err="failed to get container status \"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6\": rpc error: code = NotFound desc = could not find container \"66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6\": container with ID starting with 66d6605b1c85afe790d352fc6e4302aab6e05b6e41863d4fb1cfd0ba327cade6 not found: ID does not exist" Sep 20 02:35:17.006035 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:17.006007 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:17.023040 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:17.023013 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5747fd5c8c-cj4wn"] Sep 20 02:35:17.977854 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:17.977822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" event={"ID":"0524642c-116d-40f4-b0e1-c78a0885146d","Type":"ContainerStarted","Data":"2f6cf60efb086e3bc2637fa9ec4bc367dd9fb1c5d97e060dd18f488c95e746bf"} Sep 20 02:35:17.978258 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:17.977939 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:17.999221 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:17.999173 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" podStartSLOduration=2.591862718 podStartE2EDuration="2.999160044s" totalImagesPullingTime="407.297326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-20 02:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:35:17.996247789 +0000 UTC m=+2569.514563549" watchObservedRunningTime="2026-09-20 02:35:17.999160044 +0000 UTC m=+2569.517475802" Sep 20 02:35:19.015041 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:19.015008 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe434eba-c054-4649-8ef2-67d323b1fb41" path="/var/lib/kubelet/pods/fe434eba-c054-4649-8ef2-67d323b1fb41/volumes" Sep 20 02:35:23.431981 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:23.431937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:23.989546 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:23.989517 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58647f984c-9ghkw" Sep 20 02:35:31.914884 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:31.914840 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:38.228628 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:38.228586 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mcfb4"] Sep 20 02:35:47.615741 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:47.615712 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7_534ccdc6-9d44-4e44-97a6-ba0b734bff6c/copy-manifests/0.log" Sep 20 02:35:47.633470 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:47.633437 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7_534ccdc6-9d44-4e44-97a6-ba0b734bff6c/manager/0.log" Sep 20 02:35:48.316615 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:48.316583 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58647f984c-9ghkw_0524642c-116d-40f4-b0e1-c78a0885146d/maas-api/0.log" Sep 20 02:35:48.530154 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:48.530124 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29831190-fscfq_e9e2be10-e0b8-4660-8be3-8bc9c35061b1/cleanup/0.log" Sep 20 02:35:49.767576 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:49.767543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ttzmk_459ff040-e649-4123-abd3-4d1afdfbc599/kuadrant-console-plugin/0.log" Sep 20 02:35:50.006827 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:50.006746 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mcfb4_fd375002-74ad-475a-ab0d-f694454c9026/limitador/0.log" Sep 20 02:35:50.730307 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:50.730280 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbc588c6-8rmmb_1d7fb564-48f7-467e-8406-54554da0ac65/kube-auth-proxy/0.log" Sep 20 02:35:51.660957 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:51.660922 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wlmsv_8e36b6d3-0f46-47e0-8cdb-fa279d303b2a/main/0.log" Sep 20 02:35:52.002404 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:52.002321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gtvzr_1c12dabe-386b-4154-954b-54c081ce19be/main/0.log" Sep 20 02:35:52.337006 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:35:52.336926 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fsqjz_0811b7c1-390b-42b4-88b6-764871f70f8b/main/0.log" Sep 20 02:36:00.908124 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:00.908092 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5j9ht_cfe772d7-ed21-4135-ac2d-a2922c120101/global-pull-secret-syncer/0.log" Sep 20 02:36:01.145614 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:01.145581 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v24kh_19b4698d-5f2b-4a74-8a2f-fe9a30773c74/konnectivity-agent/0.log" Sep 20 02:36:01.218977 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:01.218884 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-50.ec2.internal_c472519add22672739b1e4cfa2793bb7/haproxy/0.log" Sep 20 02:36:06.263220 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:06.263190 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ttzmk_459ff040-e649-4123-abd3-4d1afdfbc599/kuadrant-console-plugin/0.log" Sep 20 02:36:06.342086 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:06.342057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mcfb4_fd375002-74ad-475a-ab0d-f694454c9026/limitador/0.log" Sep 20 02:36:07.682835 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.682804 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/alertmanager/0.log" Sep 20 02:36:07.704740 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.704700 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/config-reloader/0.log" Sep 20 02:36:07.724722 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.724701 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/kube-rbac-proxy-web/0.log" Sep 20 02:36:07.747670 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.747650 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/kube-rbac-proxy/0.log" Sep 20 02:36:07.771107 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.771089 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/kube-rbac-proxy-metric/0.log" Sep 20 02:36:07.792568 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.792548 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/prom-label-proxy/0.log" Sep 20 02:36:07.819637 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.819613 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c426ace4-9114-4757-b95e-aac93c12f2d3/init-config-reloader/0.log" Sep 20 02:36:07.918169 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.918136 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z9wp6_fe8a427a-0425-4b16-a503-5056dea13f18/kube-state-metrics/0.log" Sep 20 02:36:07.939823 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.939751 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z9wp6_fe8a427a-0425-4b16-a503-5056dea13f18/kube-rbac-proxy-main/0.log" Sep 20 02:36:07.965001 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:07.964983 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-z9wp6_fe8a427a-0425-4b16-a503-5056dea13f18/kube-rbac-proxy-self/0.log" Sep 20 02:36:08.207662 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.207580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vkwfk_0afd4755-1f3e-41c3-ab46-98d95e2ffb23/node-exporter/0.log" Sep 20 02:36:08.235158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.235135 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vkwfk_0afd4755-1f3e-41c3-ab46-98d95e2ffb23/kube-rbac-proxy/0.log" Sep 20 02:36:08.262948 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.262924 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vkwfk_0afd4755-1f3e-41c3-ab46-98d95e2ffb23/init-textfile/0.log" Sep 20 02:36:08.365195 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.365151 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-8s2nk_88933f47-3e2c-4e34-8850-8f529e247cb6/kube-rbac-proxy-main/0.log" Sep 20 02:36:08.395529 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.395501 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-8s2nk_88933f47-3e2c-4e34-8850-8f529e247cb6/kube-rbac-proxy-self/0.log" Sep 20 02:36:08.415992 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.415968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-8s2nk_88933f47-3e2c-4e34-8850-8f529e247cb6/openshift-state-metrics/0.log" Sep 20 02:36:08.640479 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.640452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-6dfdj_1ad72dcc-eb10-465b-9944-055f3e79c1f8/prometheus-operator/0.log" Sep 20 02:36:08.663494 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.663476 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-6dfdj_1ad72dcc-eb10-465b-9944-055f3e79c1f8/kube-rbac-proxy/0.log" Sep 20 02:36:08.729448 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.729427 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647cd948-kwzbx_2a3a1741-cebb-4a8d-a7cb-a061d02bde83/telemeter-client/0.log" Sep 20 02:36:08.748939 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.748896 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647cd948-kwzbx_2a3a1741-cebb-4a8d-a7cb-a061d02bde83/reload/0.log" Sep 20 02:36:08.769526 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.769503 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647cd948-kwzbx_2a3a1741-cebb-4a8d-a7cb-a061d02bde83/kube-rbac-proxy/0.log" Sep 20 02:36:08.800082 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.800064 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/thanos-query/0.log" Sep 20 02:36:08.819295 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.819275 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/kube-rbac-proxy-web/0.log" Sep 20 02:36:08.849642 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.849617 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/kube-rbac-proxy/0.log" Sep 20 02:36:08.872898 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.872878 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/prom-label-proxy/0.log" Sep 20 02:36:08.899113 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.899055 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/kube-rbac-proxy-rules/0.log" Sep 20 02:36:08.919938 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:08.919920 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65d66cddf-7hhml_5ab2da1e-db3a-48ad-a8da-ffdb3ac44875/kube-rbac-proxy-metrics/0.log" Sep 20 02:36:11.423677 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.423643 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9"] Sep 20 02:36:11.424084 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.424070 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe434eba-c054-4649-8ef2-67d323b1fb41" containerName="maas-api" Sep 20 02:36:11.424084 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.424085 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe434eba-c054-4649-8ef2-67d323b1fb41" containerName="maas-api" Sep 20 02:36:11.424169 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.424160 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe434eba-c054-4649-8ef2-67d323b1fb41" containerName="maas-api" Sep 20 02:36:11.427278 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.427264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.429720 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.429698 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gqx6h\"/\"kube-root-ca.crt\"" Sep 20 02:36:11.429849 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.429765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gqx6h\"/\"openshift-service-ca.crt\"" Sep 20 02:36:11.430511 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.430493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gqx6h\"/\"default-dockercfg-bbk9m\"" Sep 20 02:36:11.438215 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.438194 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9"] Sep 20 02:36:11.515468 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.515444 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-788d4985c7-24xc5_dca76d6a-b579-44ca-94b0-17b4830ee7e8/console/0.log" Sep 20 02:36:11.541415 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.541388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-proc\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.541533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.541426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjr65\" (UniqueName: \"kubernetes.io/projected/d98e59e0-083b-48f3-84b3-0bee3354d9c4-kube-api-access-sjr65\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.541533 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.541447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-lib-modules\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.541616 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.541543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-sys\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.541616 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.541571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-podres\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-sys\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642730 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-podres\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-proc\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjr65\" (UniqueName: \"kubernetes.io/projected/d98e59e0-083b-48f3-84b3-0bee3354d9c4-kube-api-access-sjr65\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-lib-modules\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642821 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-sys\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.642942 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-proc\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.643111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-podres\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.643111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.642977 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d98e59e0-083b-48f3-84b3-0bee3354d9c4-lib-modules\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.651213 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.651196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjr65\" (UniqueName: \"kubernetes.io/projected/d98e59e0-083b-48f3-84b3-0bee3354d9c4-kube-api-access-sjr65\") pod \"perf-node-gather-daemonset-lpsh9\" (UID: \"d98e59e0-083b-48f3-84b3-0bee3354d9c4\") " pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:11.737800 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:11.737684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:12.009638 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:12.009609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9"] Sep 20 02:36:12.010702 ip-10-0-137-50 kubenswrapper[2578]: W0920 02:36:12.010665 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd98e59e0_083b_48f3_84b3_0bee3354d9c4.slice/crio-1fa004d35327138ef201d6b360cb68bb37c08b8a731a5ef2ed5b5d593fa62b0c WatchSource:0}: Error finding container 1fa004d35327138ef201d6b360cb68bb37c08b8a731a5ef2ed5b5d593fa62b0c: Status 404 returned error can't find the container with id 1fa004d35327138ef201d6b360cb68bb37c08b8a731a5ef2ed5b5d593fa62b0c Sep 20 02:36:12.275897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:12.275807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" event={"ID":"d98e59e0-083b-48f3-84b3-0bee3354d9c4","Type":"ContainerStarted","Data":"8320a88b3c924dbaaf8890cff4802af1b9011d5c95b62777519c5ae3704893d1"} Sep 20 02:36:12.275897 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:12.275861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" event={"ID":"d98e59e0-083b-48f3-84b3-0bee3354d9c4","Type":"ContainerStarted","Data":"1fa004d35327138ef201d6b360cb68bb37c08b8a731a5ef2ed5b5d593fa62b0c"} Sep 20 02:36:12.276111 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:12.275937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:12.294020 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:12.293973 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" podStartSLOduration=1.293956873 podStartE2EDuration="1.293956873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-20 02:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-20 02:36:12.291365774 +0000 UTC m=+2623.809681533" watchObservedRunningTime="2026-09-20 02:36:12.293956873 +0000 UTC m=+2623.812272631" Sep 20 02:36:13.559270 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:13.559227 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wdsjv_420afa4e-3ba4-442c-8548-70cecec2bc11/dns/0.log" Sep 20 02:36:13.577967 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:13.577941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wdsjv_420afa4e-3ba4-442c-8548-70cecec2bc11/kube-rbac-proxy/0.log" Sep 20 02:36:13.643428 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:13.643392 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpl5v_6da7c700-b7cc-4c87-9400-ef17a3557825/dns-node-resolver/0.log" Sep 20 02:36:14.295945 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:14.295896 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dczpm_ee537002-c5d1-4dcf-99ae-13e1b0fe1607/node-ca/0.log" Sep 20 02:36:16.113163 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:16.113114 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbc588c6-8rmmb_1d7fb564-48f7-467e-8406-54554da0ac65/kube-auth-proxy/0.log" Sep 20 02:36:17.271736 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:17.271702 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvbhm_2313a01c-8a54-4b2e-b113-6100216ce6b5/serve-healthcheck-canary/0.log" Sep 20 02:36:17.759108 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:17.759076 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52f4v_84ca736c-c75f-4bb0-b7ad-2dbd182acc00/kube-rbac-proxy/0.log" Sep 20 02:36:17.778333 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:17.778306 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52f4v_84ca736c-c75f-4bb0-b7ad-2dbd182acc00/exporter/0.log" Sep 20 02:36:17.799271 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:17.799245 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52f4v_84ca736c-c75f-4bb0-b7ad-2dbd182acc00/extractor/0.log" Sep 20 02:36:18.291729 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:18.291704 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gqx6h/perf-node-gather-daemonset-lpsh9" Sep 20 02:36:21.002275 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:21.002235 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7_534ccdc6-9d44-4e44-97a6-ba0b734bff6c/manager/0.log" Sep 20 02:36:21.022097 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:21.022065 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59d4cf4f4c-7k8h7_534ccdc6-9d44-4e44-97a6-ba0b734bff6c/copy-manifests/0.log" Sep 20 02:36:22.581127 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:22.581056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bdfb9dff9-t6vx5_acc11168-00d0-4003-b954-9681c8a9cf77/manager/0.log" Sep 20 02:36:22.646861 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:22.646831 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-w7g5c_b27a791e-cf05-4f3a-a7f9-7f2eb8320d8e/openshift-lws-operator/0.log" Sep 20 02:36:23.024460 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:23.024433 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbc588c6-8rmmb_1d7fb564-48f7-467e-8406-54554da0ac65/kube-auth-proxy/0.log" Sep 20 02:36:25.599219 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:25.599190 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbc588c6-8rmmb_1d7fb564-48f7-467e-8406-54554da0ac65/kube-auth-proxy/0.log" Sep 20 02:36:29.321119 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.321089 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/kube-multus-additional-cni-plugins/0.log" Sep 20 02:36:29.344105 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.344077 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/egress-router-binary-copy/0.log" Sep 20 02:36:29.367969 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.367948 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/cni-plugins/0.log" Sep 20 02:36:29.392321 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.392291 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/bond-cni-plugin/0.log" Sep 20 02:36:29.414158 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.414137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/routeoverride-cni/0.log" Sep 20 02:36:29.437040 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.437018 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/whereabouts-cni-bincopy/0.log" Sep 20 02:36:29.457389 ip-10-0-137-50 kubenswrapper[2578]: I0920 02:36:29.457364 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89dck_e72130d2-cf4b-4cb7-9912-a7fe11007acd/whereabouts-cni/0.log"